Merge branch 'owen-wiki' of https://github.com/odiffey/MusareNode into polishing

.wiki/Backend_Commands.md

@@ -0,0 +1,35 @@
+# Backend Commands
+Backend commands are inputted via STDIN or if using the Utility Script by using `./musare.sh attach backend`.
+
+## Commands
+| Command | Parameters | Description |
+| --- | --- | --- |
+| `rs` | | Restart backend. |
+| `status` | | Returns all modules and a sample of information including, state, jobs queued, running and paused, concurrency (amount of jobs that can run simultaneously), and (startup) stage. |
+| `queued` | `module` | Returns all jobs queued for specified module. |
+| `running` | `module` | Returns all jobs running for specified module. |
+| `paused` | `module` | Returns all jobs paused for specified module. |
+| `jobinfo` | `UUID` | Returns a detailed overview of a specified job. |
+| `runjob` | `module job_name json_encoded_payload` | Run a specified job in a specified module including a JSON encoded payload, and return response. |
+| `eval` | `some_javascript` | Execute JavaScript within the index.js context and return response. |
+| `lockdown` | | Lockdown backend. |
+| `stats` | `module` | Returns job statistics for a specified module. |
+
+## Modules
+When specifying a module please use all lowercase. The available modules are as follows:
+
+- Cache
+- DB
+- Mail
+- Activities
+- API
+- App
+- WS
+- Notifications
+- Playlists
+- Punishments
+- Songs
+- Stations
+- Tasks
+- Utils
+- YouTube

.wiki/Configuration.md

@@ -6,24 +6,48 @@ Location: `backend/config/default.json`
 | Property | Description |
 | --- | --- |
 | `mode` | Should be either `development` or `production`. |
-| `migration` | Should be set to true if you need to update DB documents to a newer version after an update. Should be false at all other times. |
+| `migration` | Should be set to `true` if you need to update MongoDB documents to a newer version after an update. Should be false at all other times. |
 | `secret` | Set to something unique and secure - used by express's session module. |
 | `domain` | Should be the url where the site will be accessible from, usually `http://localhost` for non-Docker. |
 | `serverDomain` | Should be the url where the backend will be accessible from, usually `http://localhost/backend` for docker or `http://localhost:8080` for non-Docker. |
 | `serverPort` | Should be the port where the backend will listen on, should always be `8080` for Docker, and is recommended for non-Docker. |
 | `registrationDisabled` | If set to true, users can't register accounts. |
-| `apis.youtube.key` | Can be obtained by setting up a [YouTube API Key](https://developers.google.com/youtube/v3/getting-started). You need to use the YouTube Data API v3, and create an API key. |
-| `apis.recaptcha` | Can be obtained by setting up a [ReCaptcha Site (v3)](https://www.google.com/recaptcha/admin), or you can disable it. |
-| `apis.github` | Can be obtained by setting up a [GitHub OAuth Application](https://github.com/settings/developers). You need to fill in some values to create the OAuth application. The homepage is the homepage of frontend. The authorization callback url is the backend url with `/auth/github/authorize/callback` added at the end. For example `http://localhost/backend/auth/github/authorize/callback`. |
-| `apis.discogs` | Can be obtained by setting up a [Discogs application](https://www.discogs.com/settings/developers), or you can disable it. |
-| `smtp` | Can be obtained by setting up an SMTP server, or you can disable it. |
-| `redis.url` | Should be left alone for Docker, and changed to `redis://localhost:6379/0` for non-Docker. |
-| `redis.password` | Should be the Redis password you either put in your `startRedis.cmd` file for Windows, or `.env` for docker. |
-| `mongo.url` | Needs to have the proper password for the MongoDB musare user, and for non-Docker you need to replace `@musare:27017` with `@localhost:27017`. |
+| `hideAutomaticallyRequestedSongs` | If `true` any automatically requested songs will be hidden. |
+| `hideAnonymousSongs` | If `true` any anonymously requested songs will be hidden. |
+| `sendDataRequestEmails` | If `true` all admin users will be sent an email if a data request is received. |
+| `apis.youtube.key` | YouTube Data API v3 key, obtained from [here](https://developers.google.com/youtube/v3/getting-started). |
+| `apis.youtube.rateLimit` | Minimum interval between YouTube API requests in milliseconds. |
+| `apis.youtube.requestTimeout` | YouTube API requests timeout in milliseconds. |
+| `apis.youtube.retryAmount` | The amount of retries to perform of a failed YouTube API request. |
+| `apis.recaptcha.secret` | ReCaptcha Site v3 secret, obtained from [here](https://www.google.com/recaptcha/admin). |
+| `apis.recaptcha.enabled` | Whether to enable ReCaptcha at email registration. |
+| `apis.github.client` | GitHub OAuth Application client, obtained from [here](https://github.com/settings/developers). |
+| `apis.github.secret` | GitHub OAuth Application secret, obtained with client. |
+| `apis.github.redirect_uri` | The authorization callback url is the backend url with `/auth/github/authorize/callback` appended, for example `http://localhost/backend/auth/github/authorize/callback`. |
+| `apis.discogs.client` | Discogs Application client, obtained from [here](https://www.discogs.com/settings/developers). |
+| `apis.discogs.secret` | Discogs Application secret, obtained with client. |
+| `apis.discogs.enabled` | Whether to enable Discogs API usage. |
+| `smtp.host` | SMTP Host |
+| `smtp.port` | SMTP Port |
+| `smtp.auth.user` | SMTP Username |
+| `smtp.auth.pass` | SMTP Password |
+| `smtp.secure` | Whether SMTP is secured. |
+| `smtp.enabled` | Whether SMTP and sending emails is enabled. |
+| `redis.url` | Should be left as default for Docker installations, else changed to `redis://localhost:6379/0`. |
+| `redis.password` | Redis password. |
+| `mongo.url` | For Docker replace temporary MongoDB musare user password with one specified in `.env`, and for non-Docker replace `@musare:27017` with `@localhost:27017`. |
 | `cookie.domain` | The ip or address you use to access the site, without protocols (http/https), so for example `localhost`. |
 | `cookie.secure` | Should be `true` for SSL connections, and `false` for normal http connections. |
+| `cookie.SIDname` | Name of the cookie stored for sessions. |
 | `skipConfigVersionCheck` | Skips checking if the config version is outdated or not. Should almost always be set to false. |
 | `skipDbDocumentsVersionCheck` | Skips checking if there are any DB documents outdated or not. Should almost always be set to false. |
+| `debug.stationIssue` | If set to `true` it will enable the `/debug_station` API endpoint on the backend, which provides information useful to debugging stations not skipping, as well as capure all jobs specified in `debug.captureJobs`. 
+| `debug.traceUnhandledPromises` | Enables the trace-unhandled package, which provides detailed information when a promise is unhandled. |
+| `debug.captureJobs` | Array of jobs to capture for `debug.stationIssue`. |
+| `defaultLogging.hideType` | Filters out specified message types from log, for example `INFO`, `SUCCESS`, `ERROR` and `STATION_ISSUE`. |
+| `defaultLogging.blacklistedTerms` | Filters out messages containing specified terms from log, for example `success`. |
+| `customLoggingPerModule.[module].hideType` | Where `[module]` is a module name specify hideType as you would `defaultLogging.hideType` to overwrite default. |
+| `customLoggingPerModule.[module].blacklistedTerms` | Where `[module]` is a module name specify blacklistedTerms as you would `defaultLogging.blacklistedTerms` to overwrite default. |
 | `configVersion` | Version of the config. Every time the template changes, you should change your config accordingly and update the configVersion. |
 
 ## Frontend
@@ -31,18 +55,23 @@ Location: `frontend/dist/config/default.json`
 
 | Property | Description |
 | --- | --- |
+| `mode` | Should be either `development` or `production`. |
 | `backend.apiDomain` | Should be the url where the backend will be accessible from, usually `http://localhost/backend` for docker or `http://localhost:8080` for non-Docker. |
 | `backend.websocketsDomain` | Should be the same as the `apiDomain`, except using the `ws://` protocol instead of `http://` and with `/ws` at the end. |
 | `devServer.webSocketURL` | Should be the webpack-dev-server websocket URL, usually `ws://localhost/ws`. |
 | `devServer.port` | Should be the port where webpack-dev-server will be accessible from, should always be port `81` for Docker since nginx listens on port 80, and is recommended to be port `80` for non-Docker. |
 | `frontendDomain` | Should be the url where the frontend will be accessible from, usually `http://localhost` for docker or `http://localhost:80` for non-Docker. |
-| `recaptcha.key` | Can be obtained by setting up a [ReCaptcha Site (v3)](https://www.google.com/recaptcha/admin). |
-| `recaptcha.enabled` | Keep at false to keep disabled. |
+| `recaptcha.key` | ReCaptcha Site v3 key, obtained from [here](https://www.google.com/recaptcha/admin). |
+| `recaptcha.enabled` | Whether to enable ReCaptcha at email registration. |
 | `cookie.domain` | Should be the ip or address you use to access the site, without protocols (http/https), so for example `localhost`. |
 | `cookie.secure` | Should be `true` for SSL connections, and `false` for normal http connections. |
-| `siteSettings.logo` | Path to the logo image, by default it is `/assets/wordmark.png`. |
-| `siteSettings.siteName` | Should be the name of the site. |
+| `cookie.SIDname` | Name of the cookie stored for sessions. |
+| `siteSettings.logo_white` | Path to the white logo image, by default it is `/assets/white_wordmark.png`. |
+| `siteSettings.logo_blue` | Path to the blue logo image, by default it is `/assets/blue_wordmark.png`. |
+| `siteSettings.sitename` | Should be the name of the site. |
 | `siteSettings.github` | URL of GitHub repository, defaults to `https://github.com/Musare/MusareNode`. |
+| `messages.accountRemoval` | Message to return to users on account removal. |
+| `shortcutOverrides` | Overwrite keyboard shortcuts, for example `"editSong.useAllDiscogs": { "keyCode": 68, "ctrl": true, "alt": true, "shift": false, "preventDefault": true }`. |
 | `skipConfigVersionCheck` | Skips checking if the config version is outdated or not. Should almost always be set to false. |
 | `configVersion` | Version of the config. Every time the template changes, you should change your config accordingly and update the configVersion. |
 

.wiki/Installation.md

@@ -16,6 +16,17 @@ Musare can be installed with Docker (recommended) or without, guides for both in
 5. `cp .env.example .env` and configure as per [Configuration](./Configuration.md#Docker-Environment).
 6. `./musare.sh build`
 7. `./musare.sh start`
+8. **(optional)** Register a new user on the website and grant the admin role by running `./musare.sh admin add USERNAME`.
+
+### Fixing the "couldn't connect to docker daemon" error
+
+**Windows Only**
+
+Some people have had issues while trying to execute the `docker-compose` command.
+To fix this, you will have to run `docker-machine env default`.
+This command will print various variables.
+At the bottom, it will say something similar to `@FOR /f "tokens=*" %i IN ('docker-machine env default') DO @%i`.
+Run this command in your shell. You will have to do this command for every shell you want to run `docker-compose` in (every session).
 
 ---
 
@@ -47,6 +58,12 @@ Musare can be installed with Docker (recommended) or without, guides for both in
         - **Manual**
             1. Run `startRedis.cmd` and `startMongo.cmd` to start Redis and Mongo.
             2. Execute `cd frontend && npm run dev` and `cd backend && npm run dev` separately.
+8. **(optional)** Register a new user on the website and grant the admin role by running the following in the mongodb shell.
+    ```bash
+    use musare
+    db.auth("MUSAREDBUSER","MUSAREDBPASSWORD")
+    db.users.update({username: "USERNAME"}, {$set: {role: "admin"}})
+    ```
 
 ### Setting up MongoDB
 - **Windows Only**

.wiki/Technical_Overview.md

@@ -0,0 +1,20 @@
+# Technical Overview
+
+## Our Stack
+
+- NodeJS
+- MongoDB
+- Redis
+- Nginx (not required)
+- VueJS
+
+### Frontend
+
+The frontend is a [vue-cli](https://github.com/vuejs/vue-cli) generated, [vue-loader](https://github.com/vuejs/vue-loader) single page app, that's served over Nginx or Express. The Nginx server not only serves the frontend, but can also serve as a load balancer for requests going to the backend.
+
+### Backend
+
+The backend is a scalable NodeJS / Redis / MongoDB app. User sessions are stored in a central Redis server. All data is stored in a central MongoDB server. The Redis and MongoDB servers are replicated to several secondary nodes, which can become the primary node if the current primary node goes down.
+
+We currently only utilize 1 backend, 1 MongoDB server and 1 Redis server running for production, though it is relatively easy to expand.
+

.wiki/Value_Formats.md

@@ -0,0 +1,59 @@
+# Value Formats
+
+Every input needs validation, below is the required formatting of each value.
+
+- **User**
+    - Username
+        - Description: Any letter from a-z in any case, numbers, underscores and dashes. Must contain at least 1 letter or number.
+        - Length: From 2 to 32 characters.
+        - Regex: ```/^[A-Za-z0-9_]+$/```
+    - Name
+        - Description: Any letter from any language in any case, numbers, underscores, dashes, periods, apostrophes and spaces. Must contain at least 1 letter or number.
+        - Length: From 2 to 64 characters.
+        - Regex: ```/^[\p{L}0-9 .'_-]+$/u```
+    - Email
+        - Description: Standard email address.
+        - Length: From 3 to 254 characters.
+        - Regex: ```/^[\x00-\x7F]+@[a-z0-9]+\.[a-z0-9]+(\.[a-z0-9]+)?$/```
+    - Password
+        - Description: Must include at least one lowercase letter, one uppercase letter, one number and one special character.
+        - Length: From 6 to 200 characters.
+        - Regex: ```/^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~])[A-Za-z\d!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~]/```
+    - Ban Reason
+        - Description: Any ASCII character.
+        - Length: From 1 to 64 characters.
+        - Regex: ```/^[\x00-\x7F]+$/```
+- **Station**
+    - Name
+        - Description: Any letter from a-z lowercase, numbers and underscores.
+        - Length: From 2 to 16 characters.
+        - Regex: ```/^[a-z0-9_]+$/```
+    - Display Name
+        - Description: Any ASCII character.
+        - Length: From 2 to 32 characters.
+        - Regex: ```/^[\x00-\x7F]+$/```
+    - Description
+        - Description: Any character.
+        - Length: From 2 to 200 characters.
+- **Playlist**
+    - Display Name
+        - Description: Any ASCII character.
+        - Length: From 1 to 32 characters.
+        - Regex: ```/^[\x00-\x7F]+$/```
+- **Song**
+    - Title
+        - Description: Any ASCII character.
+        - Length: From 1 to 32 characters.
+        - Regex: ```/^[\x00-\x7F]+$/```
+    - Artists
+        - Description: Any character and not NONE.
+        - Length: From 1 to 64 characters.
+        - Quantity: Min 1, max 10.
+    - Genres
+        - Description: Any ASCII character.
+        - Length: From 1 to 32 characters.
+        - Quantity: Min 1, max 16.
+        - Regex: ```/^[\x00-\x7F]+$/```
+    - Thumbnail
+        - Description: Valid url. If site is secure only https prepended urls are valid.
+        - Length: From 1 to 256 characters.

README.md

@@ -1,84 +1,72 @@
-# MusareNode
+![Musare](frontend/dist/assets/blue_wordmark.png)
 
-Based off of the original [Musare](https://github.com/Musare/MusareMeteor), which utilized Meteor.
+# Musare
 
-MusareNode now uses NodeJS, Express, VueJS and websockets - among other technologies. We have also implemented the ability to host Musare in [Docker Containers](https://www.docker.com/).
+Musare is an open-source collaborative music listening and catalogue curation application. Currently supporting YouTube based content.
 
-The master branch is available at [musare.com](https://musare.com)
-You can also find the staging branch at [musare.dev](https://musare.dev)
+---
 
-<br />
-
-## Getting Started
+## Documentation
 - [Installation](./.wiki/Installation.md)
 - [Configuration](./.wiki/Configuration.md)
 - [Utility Script](./.wiki/Utility_Script.md)
-
-<br />
-
-## Our Stack
-
-- NodeJS
-- MongoDB
-- Redis
-- Nginx (not required)
-- VueJS
-
-### **Frontend**
-
-The frontend is a [vue-cli](https://github.com/vuejs/vue-cli) generated, [vue-loader](https://github.com/vuejs/vue-loader) single page app, that's served over Nginx or Express. The Nginx server not only serves the frontend, but can also serve as a load balancer for requests going to the backend.
-
-### **Backend**
-
-The backend is a scalable NodeJS / Redis / MongoDB app. User sessions are stored in a central Redis server. All data is stored in a central MongoDB server. The Redis and MongoDB servers are replicated to several secondary nodes, which can become the primary node if the current primary node goes down.
-
-We currently only utilize 1 backend, 1 MongoDB server and 1 Redis server running for production, though it is relatively easy to expand.
-
-<br />
-
-## Extra
-
-Below is a list of helpful tips / solutions we've collected while developing MusareNode.
-
-### Fixing the "couldn't connect to docker daemon" error
-
-Some people have had issues while trying to execute the `docker-compose` command.
-To fix this, you will have to run `docker-machine env default`.
-This command will print various variables.
-At the bottom, it will say something similar to `@FOR /f "tokens=*" %i IN ('docker-machine env default') DO @%i`.
-Run this command in your shell. You will have to do this command for every shell you want to run `docker-compose` in (every session).
-
-### Calling Toasts
-
-You can create Toast notifications using our custom package, [`toasters`](https://github.com/jonathan-grah/vue-roaster), using the following code:
-
-```js
-import Toast from "toasters";
-new Toast({ content: "Hi!", persistant: true });
-```
-
-### Set user role
-
-When setting up you will need to grant yourself the admin role, using the following commands:
-
-```bash
-docker-compose exec mongo mongo admin
-
-use musare
-db.auth("MUSAREDBUSER","MUSAREDBPASSWORD")
-db.users.update({username: "USERNAME"}, {$set: {role: "admin"}})
-```
-
-OR use the Linux script:
-
-```
-tools/linux/makeUserAdmin.sh YOUR_MUSARE_USERNAME YOUR_MONGO_MUSARE_PASSWORD
-```
-
-<br />
+- [Backend Commands](./.wiki/Backend_Commands.md)
+- [Technical Overview](./.wiki/Technical_Overview.md)
+- [Value Formats](./.wiki/Value_Formats.md)
+
+---
+
+## Features
+- **Playlists**
+    - User created playlists
+    - Automatically generated playlists for genres
+    - Privacy configuration
+    - Liked and Disliked songs playlists per user
+    - Bulk import songs from YouTube playlist
+    - Add songs from verified catalogue or YouTube
+    - Ability to download in JSON format
+- **Stations**
+    - Playlist mode to listen to selected playlists
+    - Party mode to allow other users to add songs to queue
+    - Ability to blacklist playlists to prevent songs within from playing
+    - Themes
+    - Privacy configuration
+    - Favoriting
+    - Official stations controlled by admins (playlist mode only)
+    - User created and controlled stations
+    - Pause playback just in local session
+    - Station-wide pausing by admins or owners
+    - Vote to skip songs
+    - Force skipping song by admins or owners
+    - Add songs to queue from verified catalogue or YouTube (party mode only)
+- **Song Management**
+    - Verify songs to allow them to be searched for and played in official stations
+    - Hide songs to remove from unverified catalogue
+    - Import Album (WIP) to import songs in bulk
+    - Discogs integration to import metadata
+    - Ability for users to report issues with songs and admins to resolve
+    - Configurable skip duration and song duration to cut intros and outros
+    - Request songs from YouTube in official stations or admin area
+    - Any song added to playlists or stations will be automatically requested
+- **Users**
+    - Activity logs
+    - Profile page showing public playlists and activity logs
+    - Text or gravatar profile pictures
+    - Email or Github login/registration
+    - Preferences to tailor site usage
+    - Password reset
+    - Data deletion management
+    - ActivityWatch integration
+- **Punishments**
+    - Ban users
+    - Ban IPs
+- **News**
+    - Admins can add/edit/remove news items
+    - Markdown editor
+- **Dark Mode**
+
+---
 
 ## Contact
 
-Get in touch with us via email at [core@musare.com](mailto:core@musare.com) or join our [Discord Guild](https://discord.gg/Y5NxYGP).
-
-You can also find us on [Facebook](https://www.facebook.com/MusareMusic) and [Twitter](https://twitter.com/MusareApp).
+Get in touch with us via email at [core@musare.com](mailto:core@musare.com).

backend/config/template.json

@@ -10,7 +10,6 @@
 	"hideAutomaticallyRequestedSongs": false,
     "hideAnonymousSongs": false,
 	"sendDataRequestEmails": true,
-	"fancyConsole": true,
 	"apis": {
 		"youtube": {
 			"key": "",
@@ -33,11 +32,6 @@
 			"enabled": false
 		}
 	},
-	"cors": {
-		"origin": [
-			"http://localhost"
-		]
-	},
 	"smtp": {
 		"host": "smtp.mailgun.org",
 		"port": 587,

backend/index.js

@@ -39,8 +39,6 @@ if (
 	process.exit();
 }
 
-const fancyConsole = config.get("fancyConsole");
-
 if (config.debug && config.debug.traceUnhandledPromises === true) {
 	console.log("Enabled trace-unhandled/register");
 	import("trace-unhandled/register");
@@ -95,7 +93,6 @@ class ModuleManager {
 		this.jobManager = new JobManager();
 		this.i = 0;
 		this.lockdown = false;
-		this.fancyConsole = fancyConsole;
 		this.debugLogs = {
 			stationIssue: []
 		};

tools/linux/makeUserAdmin.sh

@@ -1,12 +0,0 @@
-#!/bin/bash
-
-USERNAME=$1
-MONGO_MUSARE_PASSWORD=$2
-
-if [[ -n $USERNAME ]] && [[ -n $MONGO_MUSARE_PASSWORD ]];
-then
-        echo "Attemtping to make '$USERNAME' an admin"
-        docker-compose exec mongo mongo musare -u musare -p $MONGO_MUSARE_PASSWORD --eval "db.users.update({username: '$USERNAME'}, {\$set: {role: 'admin'}})"
-else
-        echo "Syntax: makeUserAdmin MUSARE_USERNAME MONGO_MUSARE_PASSWORD"
-fi