Newznab on Ubuntu 11.10
This guide covers installing Newznab+ on a fresh copy of Ubuntu 11.10 64-bit server and assumes a basic knowledge of using Linux via the command line.
TODO: Install and configure Memcache?
Install Ubuntu
Grab a copy of Ubuntu from their website and install it. Once the install is complete, log in, fire up a terminal window, do a full upgrade and reboot (entering your password when prompted):
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y dist-upgrade
sudo reboot
You’ll probably want to install SSH if you don’t have it already:
sudo apt-get install -y ssh
Additionally, we’re going to be compiling some code from source, so we’ll need the tools to do so:
sudo apt-get install build-essential checkinstall
Install Newznab
Once Ubuntu is installed and updated, we can begin installing Newznab. Before
we get started though, we need to decide where to install Newznab. Typically,
web-related stuff goes in /var/www/
, so lets put Newznab in
/var/www/newznab
. Let’s create the directory and set it writeable by our
user for now:
sudo mkdir -p /var/www/newznab
sudo chmod 777 /var/www/newznab
Now we can begin installing the prerequisites.
Prerequisites
Newznab as a few dependencies, so let’s start with installing the required software. According to the install docs, Newznab needs the following software:
PHP 5.2+
GD Imaging Library w/PHP integration
PEAR
Database: MySQL (or Percona)
Web Server: Apache (or Nginx)
Its worth mentioning a few things at this point. The first is that the default version of MySQL that comes with Ubuntu is kind of old (version 5.1). Therefore, we’ll also cover installing a higher-performance version (Percona version 5.5) instead. Additionally, Apache is listed as a requirement, however it is possible to use a different web server instead. Therefore, we’ll go over how to use the blazing-fast Nginx as well for those who don’t want to use Apache.
The following software is optional, but we’ll also cover installing it and setting it up:
Unrar
FFmpeg
Lame
MediaInfo
Sphinx
By the end of this guide we’ll have a fully working Newznab+ install with all optional components working as well.
PHP
Let’s start by installing PHP and required PHP extensions:
sudo apt-get install -y php5 php5-dev php-pear php5-gd php5-mysql php5-curl
The install docs also say that PHP needs a few tweaks. Using your favorite
text-editor (I’m using nano
) open up php.ini
:
sudo nano /etc/php5/cli/php.ini
Scroll down and look for register_globals
, make sure it is set to:
register_globals = Off
Next, look for max_execution_time
, and set it to (2 minutes):
max_execution_time = 120
Next, look for memory_limit
, and either set it to 1024M
, or -1
for
no limit:
memory_limit = 1024M
While you’re here, you should also set PHP’s date.timezone
(a list of
available timezone settings can be found here):
date.timezone = Europe/London
That should do it for PHP for the time being (we’ll come back to it when we configure the web server).
Database
You have a few options here, we’ll cover two:
Use MySQL 5.1
Use Percona 5.5
I’d recommend that you use Percona, but the choice is yours. Pick one and follow one of the sections below (do not install MySQL and Percona).
MySQL 5.1
If you decided to use MySQL, simply install it via aptitude
:
sudo apt-get install mysql-server-5.1 mysql-client-5.1 libmysqlclient-dev
Percona 5.5
Installing Percona requires a little extra work upfront, but in the long run it is generally worth it as Percona is a high-tuned fork of MySQL. The first step is to add Percona into aptitude:
gpg --keyserver hkp://keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
gpg -a --export CD2EFD2A | sudo apt-key add -
That retrieves and installs the GPG keys needed to add Percona’s aptitude repository. Now we need to tell aptitude about the repositories:
sudo sh -c "echo \"\n#Percona\" >> /etc/apt/sources.list"
sudo sh -c "echo \"deb http://repo.percona.com/apt lenny main\" >> /etc/apt/sources.list"
sudo sh -c "echo \"deb-src http://repo.percona.com/apt lenny main\" >> /etc/apt/sources.list"
Now update and install it:
sudo apt-get update
sudo apt-get install -y percona-server-client-5.5 \
percona-server-server-5.5 \
libmysqlclient-dev
It’ll ask for a password to use for the MySQL root user, so pick one and remember it.
Web Server
OK, now for the web server. Once again, you have multiple choices and we’ll cover two:
Apache
Nginx
Apache is “easier” and generally more flexible, but it also tends to be a system hog. So, if you can’t make up your mind it is probably best to take the easy route and install Apache. If you feel like squeezing the most out of your machine and you are a little more skilled with configuring software, Nginx might be a good choice.
Again, pick one and follow the steps below (don’t install both).
Apache
First install it:
sudo apt-get install -y apache2
You’ll also need to configure the php.ini
for Apache, so follow the section
above about configuring PHP (from above), but this time, edit the file
/etc/php5/apache2/php.ini
:
sudo nano /etc/php5/apache2/php.ini
Let’s create the site configuration file for Apache. Open up a new file:
sudo nano /etc/apache2/sites-available/newznab
Here is a template you can use for this file. You should change settings where appropriate for your setup:
<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName localhost # You might want to change this
# These paths should be fine
DocumentRoot /var/www/newznab/www
ErrorLog /var/log/apache2/error.log
LogLevel warn
</VirtualHost>
Now we need to disable the default Apache settings, enable the one we just
created for Newznab and enable modrewrite
:
sudo a2dissite default
sudo a2ensite newznab
sudo a2enmod rewrite
sudo service apache2 restart
Nginx
First install it:
sudo apt-get install -y nginx
Now, for Nginx there are multiple ways to serve PHP files. Probably the best
is php-fpm
, which is basically a daemon that runs and serves PHP to Nginx.
So, let’s go ahead and install that now:
sudo apt-get install -y php5-fpm
You’ll also need to configure the php.ini
for FPM, so follow the section
above about configuring PHP (from above), but this time, edit the file
/etc/php5/fpm/php.ini
:
sudo nano /etc/php5/fpm/php.ini
Just for good measure, restart the daemon:
sudo /etc/init.d/php5-fpm restart
Let’s create the configuration file for Nginx. Open a new file:
sudo nano /etc/nginx/sites-available/newznab
Here is a template you can use for this file. You should change settings where appropriate for your setup:
server {
# Change these settings to match your machine
listen 80 default_server;
server_name localhost;
# Everything below here doesn't need to be changed
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
root /var/www/newznab/www/;
index index.html index.htm index.php;
location ~* \.(?:ico|css|js|gif|inc|txt|gz|xml|png|jpe?g) {
expires max;
add_header Pragma public;
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
}
location / { try_files $uri $uri/ @rewrites; }
location @rewrites {
rewrite ^/([^/\.]+)/([^/]+)/([^/]+)/? /index.php?page=$1&id=$2&subpage=$3 last;
rewrite ^/([^/\.]+)/([^/]+)/?$ /index.php?page=$1&id=$2 last;
rewrite ^/([^/\.]+)/?$ /index.php?page=$1 last;
}
location /admin { }
location /install { }
location ~ \.php$ {
include /etc/nginx/fastcgi_params;
fastcgi_pass 127.0.0.1:9000;
# The next two lines should go in your fastcgi_params
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
}
Let’s make sure that the Nginx logs directory exists and is writeable:
sudo mkdir -p /var/log/nginx
sudo chmod 755 /var/log/nginx
Now, let’s disable the default
Nginx site handler and enable our
newznab
configuration:
sudo unlink /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/newznab /etc/nginx/sites-enabled/newznab
Finally, let’s restart Nginx:
sudo service nginx restart
Extras
unrar
Unrar is used to extract files from releases. Installing unrar is trivial:
sudo apt-get install -y unrar
FFmpeg
Once again, the version of FFmpeg in Ubuntu’s repository is old and crusty, so lets build a newer one and also add x264 support. Start by grabbing the source code we need:
wget http://ffmpeg.org/releases/ffmpeg-0.8.7.tar.gz
wget ftp://ftp.videolan.org/pub/videolan/x264/snapshots/last_stable_x264.tar.bz2
sudo apt-get install -y libfaac-dev libjack-jackd2-dev \
libmp3lame-dev libopencore-amrnb-dev \
libopencore-amrwb-dev libsdl1.2-dev libtheora-dev \
libva-dev libvdpau-dev libvorbis-dev libx11-dev \
libxfixes-dev texi2html yasm zlib1g-dev \
libdirac-dev libxvidcore-dev
Let’s build x264
first:
mkdir x264
tar --strip-components=1 -jxf last_stable_x264.tar.bz2 -C x264
cd x264
./configure --enable-static
make
sudo checkinstall --pkgname=x264 --pkgversion="3:$(./version.sh | \
awk -F'[" ]' '/POINT/{print $4$5}')" \
--backup=no --deldoc=yes --fstrans=no --default
Now, onto FFmpeg
:
cd ../
tar xvfz ffmpeg-0.8.7.tar.gz
cd ffmpeg-0.8.7/
./configure --enable-gpl --enable-libfaac --enable-libmp3lame \
--enable-libopencore-amrnb --enable-libopencore-amrwb \
--enable-libtheora --enable-libvorbis --enable-libx264 \
--enable-nonfree --enable-postproc --enable-version3 \
--enable-x11grab --enable-libdirac --enable-libxvid
make
sudo checkinstall --pkgname=ffmpeg --pkgversion="5:0.8.7" --backup=no \
--deldoc=yes --fstrans=no --default
hash x264 ffmpeg ffplay ffprobe
Lame
Lame is used for processing audio samples:
sudo apt-get install -y lame
MediaInfo
MediaInfo is used to gain information about various types of media files found
in releases. In order to install it we simply need to add the MediaInfo
aptitude
repository:
sudo apt-get install -y python-software-properties
sudo add-apt-repository ppa:shiki/mediainfo
sudo apt-get update
sudo apt-get install -y mediainfo
Sphinx
Sphinx is used for full-text searching. It is insanely fast and if you really
want your Newznab+ install to fly, it is highly recommended. Unfortunately the
version in Ubuntu’s aptitude
repository is horribly old, so we’ll need to
build a newer version.
Let’s download the source and extract it:
wget http://sphinxsearch.com/files/sphinx-2.0.2-beta.tar.gz
tar xvfz sphinx-2.0.2-beta.tar.gz
cd sphinx-2.0.2-beta
Download and extract libstemmer_c
:
wet http://snowball.tartarus.org/dist/libstemmer_c.tgz
tar --strip-components=1 -zxf libstemmer_c.tgz -C libstemmer_c
Configure it:
./configure --prefix=/usr/local --with-libstemmer
Now we’re ready to compile Sphinx. For this step you can speed up the
compilation on a multi-core system. If you have a 4-core system, for example,
you can do (replace j4
with the number of cores your machine has):
make -j4
Once that finally finished, install it:
sudo checkinstall --pkgname=sphinx --pkgversion="2.0.2-beta" --backup=no \
--deldoc=yes --fstrans=no --default
Now we have a nice new version of Sphinx installed in /usr/local
. Binaries
are installed in /usr/local/bin
.
Newznab Source
Finally, we can now begin installing Newznab! We’ll be grabbing the latest and greatest version, so we’ll need Subversion installed first:
sudo apt-get install -y subversion
Now we can check out the Newznab code:
svn co svn://svn.newznab.com/nn/branches/nnplus /var/www/newznab
At this point we might as well set the permissions on a couple of directories as well:
sudo chmod 777 /var/www/newznab/www/lib/smarty/templates_c
sudo chmod 777 /var/www/newznab/www/covers/movies
sudo chmod 777 /var/www/newznab/www/covers/anime
sudo chmod 777 /var/www/newznab/www/covers/music
sudo chmod 777 /var/www/newznab/www
sudo chmod 777 /var/www/newznab/www/install
sudo chmod 777 /var/www/newznab/nzbfiles/
Configure Newznab
Now that Newznab is installed, we need to configure it. Configuration is done via a web browser.
Run Installer
It is now time to configure Newznab. This is done via a web-based installer.
Open up http://localhost/install
in a web browser (or whatever the
address/IP address is of your server) and follow the guided steps.
Enable Groups
Head over to /admin/group-list.php
in your web browser and pick some groups
to index by clicking “activate” on a few groups.
Set Paths and Options
Make sure to set your newznab ID
.
We need to let Newznab know where all the extra software is that we installed
earlier, so head over to /admin/site-edit.php
in your browser, scroll down
to the “3rd Party Application Paths” section and update the fields:
- Unrar Path
/usr/bin/unrar
- Mediainfo Path
/usr/bin/mediainfo
- Ffmpeg Path
/usr/local/bin/ffmpeg
- Lame Path
/usr/bin/lame
If you’d like to enable audio previews, check Save audio preview
. It is
worthwhile to do rar checking, so set Check For Passworded Releases
to
Deep
.
That’s it for configuration right now, but don’t close your browser yet as we’ll be coming back to the configuration page when configuring Sphinx.
Indexing
Back at the command line its time to fire up the binaries and releases update scripts:
cd /var/www/newznab/misc/update_scripts
The update_scripts
folder contains a lot of scripts. The most important
ones are update_binaries.php
and update_releases.php
. If you have any
experience with screen
or tmux
, it is highly recommended that you use
one of these to run the update scripts, as it will allow you to monitor the
update process, observe and resolve issues; this is especially important for
newcomers to Newznab. With that said, Newznab also ships with an init-sytle
script that can be installed to make Newznab run more or less as a daemon that
will start and stop with startup and shutdown, respectively.
Screen or tmux
If you want to go the screen
or tmux
route, you’ll need to pick one and
install it:
# Install screen...
sudo apt-get install -y screen
# ...or tmux
sudo apt-get install -y tmux
In the nix_scripts
directory there is a useful script called
newznab_screen.sh
that runs update_binaries.php
and
update_releases.php
, in addition to a few other scripts, continuously and
automatically. First, we need to modify it however, so lets change dir and
make a copy:
cd /var/www/newznab/misc/update_scripts/nix_scripts
cp newznab_screen.sh newznab_screen_local.sh
Now open newznab_screen_local.sh
in a text editor and modify
NEWZNAB_PATH
near the top to point to our installation path:
nano newznab_screen_local.sh
Set NEWZNAB_PATH
:
export NEWZNAB_PATH="/var/www/newznab/misc/update_scripts"
Now we can run the script via screen
:
screen sh newznab_screen_local.sh
You should see the script download headers for the groups that you have enabled
and then run various stages that will attempt to group and catalogue the
headers. For now, just leave the script running and detach from screen
by
typing cntl a d
.
Sphinx
As mentioned previously, Sphinx is a fast full-text indexer. By default, it is
disabled in Newznab, so go ahead and enable it by visiting
/admin/site-edit.php
and setting “Use Sphinx” to “Yes”. While there,
you’ll also notice that there are a few other configuration options for Sphinx.
By default, Sphinx will index all of the release information, however, there are
three other optional indexes: NZB contents, NFO contents and release files.
Enabling these optional indexes will add increased processing time, so you will
likely want to experiment to see what combination works best for your hardware.
For now, you don’t have to enable any of the optional indexes.
To get Sphinx running, we need to generate a sphinx.conf
file. To do this
we’ll use the nnindexer.php
script in misc/sphinx
:
cd /var/www/newznab/misc/sphinx
./nnindexer generate
The script will print out the location of the sphinx.conf
file, which by
default will be /var/www/newznab/sphinx/sphinx.conf
. This path needs to be
entered into the “Sphinx Configuration Path” setting located at
/admin/site-edit.php
.
Now we need to start the search daemon and create the indexes and restart the daemon:
./nnindexer.php daemon
./nnindexer.php index full all
./nnindexer.php index delta all
./nnindexer.php daemon --stop
./nnindexer.php daemon
Summary of Installed Software
PHP v5.3.6
Pear v1.9.2
MySQL v5.1.58 or
Percona v5.5.17
Apache v2.2.20 or
Nginx 1.0.5
FFmpeg v0.8.7
MediaInfo v0.7.50
Lame v3.98.4
unrar v4.00-beta3
Sphinx 2.0.2-beta