Myrient Search Engine

It is finally here. There is now a way to search all of Myrient's offerings. Myrient Search can be accessed by clicking the link.

Resource Requirements

• 1.5GB-ish of memory for the initial crawl (can be reduced by tweaking environment variables at the cost of slower indexing)
• 800MB-ish of memory for running the server
• HTTPS for some CORS functions to work correctly.

Post Metasearch update

Please clear your elasticsearch instance, and possibly run a new file rebuild to ensure there are no errors after updating your docker container or pulling the repo.

Self-Hosting

Docker Method (Recommended)

Requirements

• Docker / Docker Compose

Instructions

1. Download the docker-compose.yml file
2. Start the server with docker-compose up -d

Manual Method (Development)

Requirements

• nodejs
• npm
• PostgreSQL
• Elasticsearch
• Docker (optional)

Instructions

1. Clone the repository. git clone https://github.com/alexankitty/Myrient-Search-Engine
2. Install dependencies. npm i
3. Run your PostgreSQL and Elasticsearch instances. A docker compose file is provided in the repository for convenience.
4. Start the server. node server.js

HTTPS Encryption

Use something like nginx and add a site to sites-available called myrient-search in /etc/nginx/sites-available. Link the site to the sites-enabled folder. ln -sf /etc/nginx/sites-available/myrient-search /etc/nginx/sites-enabled/myrient-search

nginx Site Template

server {
    listen 80;
    listen [::]:80;

    server_name server address.tld
    root /usr/share/nginx
    access_log on;
}

#server {

    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name serveraddress.tld;
    access_log on;

    root /usr/share/nginx;

    location / {
        add_header Cache-Control no-cache;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:8062/;
    }

}

SSL Certificate

For the SSL certificate you can use certbot via the certbot -d servername.tld command and adding it to your crontab. Additional Information for Certbot Setup

Opengraph

To ensure OpenGraph metadata embed for chat apps works correctly, please be sure to set HOSTNAME in .env or docker-compose.yml to the FQDN (fully qualifed domain name) of the server that is hosting the site.

Metadata

To enable metadata synchronize and matching, you will need to create a developer application in the Twitch TV Developer Console and then add your client id to TWITCH_CLIENT_ID in .env or docker-compose.yml along with adding your client secret to TWITCH_CLIENT_SECRET. Metadata takes about half an hour to synchronize from IGDB to your database, and about another half an hour to match via Postgres Full Text Search. Once all other database maintenance operations are done, the database will attempt to match anything that still isn't matched using a much slower fuzzy trigram search that can take up to a day to complete. These processes won't run again until a new crawl of myrient has been performed and the file count has increased.

Contributing

You know the usual fluff.
Is there a missing category or string association? lib/categories.json and any of the files under lib/json/relatedkeywords can both updated to include these. If you do update/improve these, please put in a pull request so that it can be added to the public hosted server, as well.
Pull requests are always welcome. Make sure to make any changes clear in your pull request, and if possible, run any files you've modified through prettier.