Open Source Projects Europe

I've gotten uprcl to work except for populating the Artist, I will rebuild and try again with latest source.

Arch needed a few additional dependencies, perhaps some are optional...
depends=('libmpdclient' 'libupnpp>=0.15.3' 'libmicrohttpd' 'jsoncpp' 'curl' 'expat' 'python-requests' 'recoll' 'python-bottle' 'python-mutagen' 'aspell-en' 'id3lib' 'python-waitress' 'sqlite3' )

I still don't see /dev/shm being used, but maybe it's not needed for uprcl?

Thanks again for upmpdcli

Skunark

Thanks for reporting this !

The shm error message is actually not significant, the segment is only used in conjunction with OpenHome Credentials service, and never created if this not used. I just pushed a modification to silence it, it also ennoyed me every time...

I think that the AUR package would need an update, but I'm not maintaining it (not running Arch myself). The reference should be the Debian package, which I try to keep in good shape in upmpdcli/debian. See the 'control' file for the dependancies. The package is split into the base upmpdcli renderer and the different media server modules, because not everybody needs everything. As far as I can see, the Debian deps match what you unfortunately had to find out by yourself...

Please keep me updated about the issue with the Artist field. uprcl is still quite experimental at this point. It aims to emulate the excellent but not open-source minimserver so as to provide a replacement for it should it go away. Not quite there yet :)

I did let the package maintainer know on AUR of the dependencies for uprcl; but yeah, I should have thought of looking into the Debian directory, but it wasn't many. :) I will dig into the tag issues this week. Most of my songs have been ripped via python audiotools and ID3 tag and album art updates with Beets and musicbrainz. I will reduce the amount of files uprcl scans and try other files if needed. Should I expect, artist, album, genre and album art to all work?

I skimmed some of the uprcl files and noticed this:

./src/mediaserver/cdplugins/uprcl/uprcltags.py:if __name__ == '__main__':
./src/mediaserver/cdplugins/uprcl/uprcltags.py-    confdir = "/home/dockes/.recoll-mp3"

Is there anything useful in .recoll-mp3 to make this work?

I don't think that you "should have thought" of looking into the debian directory. I was just mentionning it because it's a useful reference.

About the .recoll-mp3 file: uprcl creates its own Recoll config file in /var/cache/upmpdcli/uprcl, so the one I use for testing is irrelevant.

What recoll version are you using ?

The generation of fields is quite a complicated pipe. The first step is inside Recoll, so you could have a look at it by using recoll itself. "recoll -c /var/cache/upmpdcli/uprcl", then search for any term, hit Preview on a result, right-click the preview and select "Show fields". This will show you what Recoll has extracted and stored. You may need to adjust the permissions on the upmpdcli/uprcl directory because it belongs to upmpdcli and you need to read things in there.

You could also dump the ID3 data for one of the songs and attach it here, maybe it will give me an idea of what goes wrong.

Also, thanks for the very generous donation !

Recoll(1.24.1) seems to have the right information, or at least to me everything looked fine. Attached is the recoll output (notes.txt), mutagen-inspect output and the uprcl configuration settings.

Neither Lumin nor Kazoo shows the Artist or Genre, but the "Albums" are listed.

I'm very thankful of this! Many drinks are owed here :)

Also the log (log_level=5) output when asking lumin to list Artists:

<u:Browse xmlns:u="urn:schemas-upnp-org:service:ContentDirectory:1">
<ObjectID>0$uprcl$=Artist</ObjectID>
<BrowseFlag>BrowseDirectChildren</BrowseFlag>
<Filter>*</Filter>
<StartingIndex>0</StartingIndex>
<RequestedCount>30</RequestedCount>
<SortCriteria></SortCriteria>
</u:Browse>

:4:src/mediaserver/contentdirectory.cxx:337::ContentDirectory::actBrowse:  ObjectID 0$uprcl$=Artist BrowseFlag BrowseDirectChildren Filter * StartingIndex 0 RequestedCount 30 SortCriteria 
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$albums$1
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$=Composer
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$folders$d4
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$items
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$=Artist
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$albums$2
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$=Date
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$=Comment
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$=Genre
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$folders
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$=Conductor
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$folders$d3
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$untagged
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$albums
:5:src/mediaserver/cdplugins/plgwithslave.cxx:487::ContentCache::purge: erasing uprcl:0$uprcl$
:5:src/mediaserver/cdplugins/plgwithslave.cxx:505::ContentCache::get: not found uprcl:0$uprcl$=Artist

:4:src/mediaserver/cdplugins/cmdtalk-fixed.cpp:140::CmdTalk: Got empty line

:5:src/mediaserver/cdplugins/plgwithslave.cxx:351::PlgWithSlave::results: got 0 entries 
:5:src/mediaserver/cdplugins/plgwithslave.cxx:511::ContentCache::set: uprcl:0$uprcl$=Artist
:5:src/mediaserver/cdplugins/plgwithslave.cxx:441::searchCacheEntryToResult: filter  start 0 cnt 30 res.size 0

This is weird, I can't reproduce it of course, it would be too easy :)

The recoll fields seem normal, so we can probably eliminate a recoll or mutagen issue.

The Album tree is special, it's not built like Artist/Genre etc. I guess that you don't see anything in Comment, Conductor, Date, Group and Orchestra either ?

When upmpdcli starts up, the directory is built from the recoll index data. You should see messages like the following while this happens (or they are in the log):

uprcl: Indexing took 1.50 Seconds
uprcl: Estimated alldocs query results: 14324
uprcl: Retrieved 24931 docs in 3.42 Seconds
uprcl: _rcl2folders took 1.12 Seconds
uprcl: recolltosql: processed 24931 docs in 6.52 Seconds
uprcl: Init done

Do you see any other messages during this phase (esp. between _rcl2folders and recolltosql) ?

Thanks for your help in debugging this by the way. I always want to come back to this module, but there is always something to prevent me from doing so.

The other fields are not populated as well, i've attached the systemd log (fresh.log), looks like it does match what you described. I removed the uprcl cache directory before starting the daemon.

I also attached the upmpdcli.log for the same duration.

I'm still looking through the various logs, not sure if this is helpful

Nov 06 22:15:11 hostname upmpdcli[14194]: uprcl: Browse root: rootmap now {'0$uprcl$folders': 'folders', '0$uprcl$untagged': 'untagged', '0$uprcl$albums': 'tags', '0$uprcl$items': 'tags', '0$uprcl$=Artist': 'tags', '0$uprcl$=Comment': 'tags', '0$uprcl$=Composer': 'tags', '0$uprcl$=Conductor': 'tags', '0$uprcl$=Date': 'tags', '0$uprcl$=Genre': 'tags', '0$uprcl$=Group': 'tags', '0$uprcl$=Orchestra': 'tags'}
Nov 06 22:15:11 hostname upmpdcli[14194]: CMDTALK: uprcl-app.py: browse: {'cmdtalk:proc': 'browse', 'flag': 'meta', 'objid': '0$uprcl$=Artist'}. g_httphp [192.168.86.27:9090]
Nov 06 22:15:11 hostname upmpdcli[14194]: CMDTALK: uprcl-app.py: browse: idpath: <=Artist>
Nov 06 22:15:11 hostname upmpdcli[14194]: CMDTALK: uprcl-app.py: processmessage: processor raised: [uprcl-app: browse: can't browse meta for now]
Nov 06 22:15:11 hostname upmpdcli[14194]: Traceback (most recent call last):
Nov 06 22:15:11 hostname upmpdcli[14194]:   File "/usr/share/upmpdcli/cdplugins/pycommon/cmdtalk.py", line 172, in processmessage
Nov 06 22:15:11 hostname upmpdcli[14194]:     outfields = processor.process(params)
Nov 06 22:15:11 hostname upmpdcli[14194]:   File "/usr/share/upmpdcli/cdplugins/pycommon/cmdtalkplugin.py", line 53, in process
Nov 06 22:15:11 hostname upmpdcli[14194]:     return self.dispatcher.run(params[prcnmkey], params)
Nov 06 22:15:11 hostname upmpdcli[14194]:   File "/usr/share/upmpdcli/cdplugins/pycommon/cmdtalkplugin.py", line 36, in run
Nov 06 22:15:11 hostname upmpdcli[14194]:     return func(params)
Nov 06 22:15:11 hostname upmpdcli[14194]:   File "/usr/share/upmpdcli/cdplugins/uprcl/uprcl-app.py", line 114, in browse
Nov 06 22:15:11 hostname upmpdcli[14194]:     raise Exception("uprcl-app: browse: can't browse meta for now")
Nov 06 22:15:11 hostname upmpdcli[14194]: Exception: uprcl-app: browse: can't browse meta for now
Nov 06 22:15:11 hostname upmpdcli[14194]: CMDTALK: uprcl-app.py: browse: {'cmdtalk:proc': 'browse', 'flag': 'children', 'objid': '0$uprcl$'}. g_httphp [192.168.86.27:9090]
Nov 06 22:15:11 hostname upmpdcli[14194]: CMDTALK: uprcl-app.py: browse: idpath: <>
Nov 06 22:15:12 hostname upmpdcli[14194]: uprcl: tags:browse: idpath <=Artist>
Nov 06 22:15:12 hostname upmpdcli[14194]: CMDTALK: uprcl-app.py: browse: {'cmdtalk:proc': 'browse', 'flag': 'children', 'objid': '0$uprcl$=Artist'}. g_httphp [192.168.86.27:9090]
Nov 06 22:15:12 hostname upmpdcli[14194]: CMDTALK: uprcl-app.py: browse: idpath: <=Artist>
Nov 06 22:15:12 hostname upmpdcli[14194]: uprcl: tagsbrowse. pid 0$uprcl$=Artist qpath ['=Artist']
Nov 06 22:15:12 hostname upmpdcli[14194]: uprcl: tagsbrowse: executing <SELECT artist.artist_id, artist.value FROM tracks, artist WHERE tracks.artist_id = artist.artist_id GROUP BY tracks.artist_id ORDER BY value> values []

The select data looks fine, the most probable issue here is that the table is empty.

I am attaching a modified uprcltags.py (/usr/share/upmpdcli/cdplugins/uprcl/uprcltags.py), with some traces added. Please install it and just restart upmpdcli (no need to erase the cache, the SQLite db is not cached, it will be rebuilt in memory from the recoll index).

I am interested in the traces from the plugin, they will go to stderr (either start upmpdcli from the command line as sudo upmpdcli -c /etc/upmpdcli.conf > /tmp/trace 2>&1, or use systemctl/journalctl for restarting and looking at the traces).

this is what /tmp/trace captured

uprcl: Path translation: pthstr: /srv/media/Music/twenty one pilots:/srv/media/Music/twenty one pilots
uprcl: runbottle: host mclaren.local port 9090 pthstr /srv/media/Music/twenty one pilots:/srv/media/Music/twenty one pilots pathprefix /uprcl
uprcl: Creating/updating index in /var/cache/upmpdcli/uprcl for "/srv/media/Music/twenty one pilots"
uprcl: runbottle: adding route for: /srv/media/Music/twenty one pilots/<filepath:path>
uprcl: _update_index: initrunning set
Bottle v0.12.13 server starting up (using WaitressServer())...
Listening on http://mclaren.local:9090/
Hit Ctrl-C to quit.

:3:common/rclinit.cpp:312::Configuration directory: /var/cache/upmpdcli/uprcl
:3:index/recollindex.cpp:667::recollindex: changing current directory to [/tmp]
:3:index/recollindex.cpp:700::recollindex: starting up
:3:rcldb/rcldb.cpp:1921::Db::waitUpdIdle: total xapian work 0 mS
:3:index/fsindexer.cpp:240::fsindexer index time:  0 mS
:3:./utils/workqueue.h:192::DbUpd: tasks 0 nowakes 0 wsleeps 1 csleeps 0
:2:./utils/workqueue.h:153::WorkQueue::waitIdle:DbUpd: not ok
:3:rcldb/rcldb.cpp:1921::Db::waitUpdIdle: total xapian work 0 mS
:3:rcldb/rcldb.cpp:1921::Db::waitUpdIdle: total xapian work 10 mS
:3:./utils/workqueue.h:192::DbUpd: tasks 0 nowakes 0 wsleeps 1 csleeps 0
:3:./utils/workqueue.h:192::Internfile: tasks 16 nowakes 15 wsleeps 14 csleeps 2
:3:./utils/workqueue.h:192::Split: tasks 0 nowakes 0 wsleeps 3 csleeps 0
uprcl: Indexing took 0.52 Seconds
:3:common/rclinit.cpp:312::Configuration directory: /var/cache/upmpdcli/uprcl
uprcl: Estimated alldocs query results: 16
uprcl: Retrieved 16 docs in 0.01 Seconds
uprcl: _rcl2folders took 0.00 Seconds
uprcl: recolltosql: processed 16 docs in 0.00 Seconds
uprcl: Init done

I did double check to make sure the file was correctly copied
md5sum /usr/share/upmpdcli/cdplugins/uprcl/uprcltags.py
e68c3e5ba93b257cb5109c53a64d8815 /usr/share/upmpdcli/cdplugins/uprcl/uprcltags.py

package versions if this provides any insight...

jsoncpp 1.8.4-2
libmicrohttpd 0.9.59-1
libmpdclient 2.16-1
libupnpp 0.17.0-3
mutagen 1.41.1-1
python-bottle 0.12.13-2
python-mutagen 1.41.1-1
python-requests 2.20.0-1
python-requests-file 1.4.3-2
python-waitress 1.1.0-2
recoll 1.24.1-2
sqlite 3.25.2-1

Ok, this means that the method where I put the traces is not called. More traces in file attached. We're getting close I think :)

I've attached the trace for the latest uprcltags.py

small snippet of the trace:

uprcl: Looking for artist (<class 'str'>) in doc
uprcl: artist not found

It seems that only the standard Recoll stored fields are found, and none of those which are specified by the configuration in /var/cache/upmpdcli/uprcl/fields. This is particularly weird, because the fields (e.g. artist) were indeed present in the recoll preview/fields that you uploaded earlier.

Could you attach a tar or zip of /var/cache/upmpdcli/uprcl ? If this is too big (I don't know the limit for attachments here), maybe upload it somewhere ?

I've attached the uprcl directory

The uprcl directory is missing the "fields" file, which defines which data should be stored in the index. This file is normally copied from /usr/share/upmpdcli/cdplugins/uprcl/rclconfig-fields during the initial run (if /var/cache/upmpdcli/uprcl does not exist).

So please try to delete /var/cache/upmpdcli/uprcl, then restart upmpdcli, and check if the fields file is indeed copied. If not there should be messages in the log, maybe there is a permission issue (source not readable by upmpdcli?)

The weird thing is that the preview fields that you copied previously from recoll should not have been possible in the absence of an appropriate "fields" file. Anyway... let's try the above.

That did the trick!

I didn't see an error message of the missing file and mostly i've been doing:
rm -rf /var/cache/upmpdcli/uprcl/*
which doesn't copy over the missing file on restart, and not
rm -rf /var/cache/upmpdcli/uprcl
which does copy over the missing file on restart. My bad

The archlinux user PKGBUILD file doesn't create the cache directory, which might have alleviated the problem. I have suggested that fix for the package maintainer.

Thank you so much for helping me figure this out. My swarm of Raspberry PIs can now access both my ripped CDs and Tidal in a single app.

• status: open --> closed
• milestone: -->

Thanks for your patient help in tracking this ! I am going to change the code to avoid this situation.

Don't forget that minimserver is still a vastly superior media server for now :)