GNU bug report logs - #47260
Package GNU MediaGoblin as a Guix service

Previous Next

To reply to this bug, email your comments to 47260 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: bug-guix <at> gnu.org
Cc: cwebber <at> dustycloud.org,
 Léo Le Bouter <lle-bout <at> zaclys.net>
Subject: Package GNU MediaGoblin as a Guix service
Date: Fri, 19 Mar 2021 23:20:33 +1100

[Message part 1 (text/plain, inline)]

This is a "meta" bug to keep track of the progress of packaging GNU
MediaGoblin, a platform for publishing images/audio/video etc. See
https://mediagoblin.org/

We have a guix-env.scm in the upstream source which should always have
the latest copy of our packaging progress and instructions to run it:

https://git.savannah.gnu.org/cgit/mediagoblin.git/tree/guix-env.scm

Current plan is:

1. Add OGG support to libsndfile which is needed to package
python-soundfile [patch 47210]

2. Package python-soundfile (see above). After this the test suite
should pass 100% with pytest installed from PyPI [patch 47181]

3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
in Guix seem to be incompatible. After this our test suite should run
100% with only dependencies from Guix!

4. Package MediaGoblin itself. The build process is ./configure/make
which is a bit weird for a Python project.

5. Get a basic Guix service working, with sqlite3 and without the
offloaded media transcoding currently using Celery/RabbitMQ.

6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
improve the no-bundled-JavaScript video/audio playing experience.

7. Work out why H264 support is missing.

8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
implicated in many bugs anyway, so there may benefits to the project to
doing this anyway.

9. Figure out how to deal with translations.

10. Add a PostgreSQL database to the Guix service instead of sqlite3.

11. We win. Maybe :)

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: "jgart" <jgart <at> dismail.de>
To: 47260 <at> debbugs.gnu.org
Subject: Re: Package GNU MediaGoblin as a Guix service
Date: Fri, 19 Mar 2021 15:50:19 +0000

This sounds like a great project. I would love MediaGoblin to be in Guix also.

> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
> improve the no-bundled-JavaScript video/audio playing experience.

What are your thoughts on rewriting the jquery? 

Should MediaGoblin be using vanilla javascript instead?

Some other possibilities could be purescript (https://www.purescript.org) or mint (http://mint-lang.com), although mint and crystal are not in guix yet and mint uses preact (http://preactjs.com) as its' runtime since 0.8.0 (https://www.mint-lang.com/blog/mint-0.8.0).

Message #11 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: "jgart" <jgart <at> dismail.de>
Cc: 47260 <at> debbugs.gnu.org
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Mon, 22 Mar 2021 10:28:34 +1100

On Fri, 19 Mar 2021, jgart wrote:

> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>
>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>> improve the no-bundled-JavaScript video/audio playing experience.
>
> What are your thoughts on rewriting the jquery? 
>
> Should MediaGoblin be using vanilla javascript instead?
>
> Some other possibilities could be purescript (https://www.purescript.org) or
> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
> (https://www.mint-lang.com/blog/mint-0.8.0).

Many of the functions we used to use jQuery for are now built into most
browsers from the last 10 years.

By far the most common are element selectors like:

  var panel = $('#header-panel')
  var arrow = $('.arrow')

which we just change to:

  var panel = document.querySelector('#header-panel')
  var arrow = document.querySelector('.arrow')

This is all vanilla JavaScript in individual files imported via <script>
without any sort of compilation process. This seems to be the simplest
path to get into distributions like Guix and Debian. The main problem
with purescript and others is that they require more dependencies, build
steps and tooling and are known by less people.

We have a few dependencies which will be harder to deal with such as the
video, audio and 3D model viewers. In the first instance, I think the
answer will be to do without this JavaScript entirely. That should be
feasible for video and audio given modern browser support. Perhaps we
don't ship 3D model support out of the box in distros for now.

Regards,
Ben

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Dr. Arne Babenhauserheide" <arne_bab <at> web.de>
To: Ben Sturmfels <ben <at> sturm.com.au>
Cc: bug-guix <at> gnu.org, 47260 <at> debbugs.gnu.org, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Mon, 22 Mar 2021 08:02:53 +0100

[Message part 1 (text/plain, inline)]

Ben Sturmfels via Bug reports for GNU Guix <bug-guix <at> gnu.org> writes:

> On Fri, 19 Mar 2021, jgart wrote:
>
>> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>>
>>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>>> improve the no-bundled-JavaScript video/audio playing experience.
>>
>> What are your thoughts on rewriting the jquery? 
>>
>> Should MediaGoblin be using vanilla javascript instead?
>>
>> Some other possibilities could be purescript (https://www.purescript.org) or
>> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
>> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
>> (https://www.mint-lang.com/blog/mint-0.8.0).
>
> Many of the functions we used to use jQuery for are now built into most
> browsers from the last 10 years.
>
> By far the most common are element selectors like:
>
>   var panel = $('#header-panel')
>   var arrow = $('.arrow')
>
> which we just change to:
>
>   var panel = document.querySelector('#header-panel')
>   var arrow = document.querySelector('.arrow')
>
> This is all vanilla JavaScript in individual files imported via <script>
> without any sort of compilation process. This seems to be the simplest
> path to get into distributions like Guix and Debian. The main problem
> with purescript and others is that they require more dependencies, build
> steps and tooling and are known by less people.
>
> We have a few dependencies which will be harder to deal with such as the
> video, audio and 3D model viewers. In the first instance, I think the
> answer will be to do without this JavaScript entirely. That should be
> feasible for video and audio given modern browser support. Perhaps we
> don't ship 3D model support out of the box in distros for now.

If you need support for m3u-playlists, you can use the player I wrote
here: https://www.draketo.de/software/m3u-player
→ https://www.draketo.de/software/m3u-player.js (save as utf-8)
(that m3u-playlists aren’t supported out of the box in most players is a
strange oversight, the code adds it for video- and audio-tags, License:
GPLv2 or later — just ask me if you need something else)

There’s also an enhanced version for Freenet, but that has lots of
performance-changes to work over high-latency networks and with paranoid
CSP-settings:
https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[signature.asc (application/pgp-signature, inline)]

Message #20 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Lemmer Webber <cwebber <at> dustycloud.org>
To: Ben Sturmfels <ben <at> sturm.com.au>
Cc: bug-guix <at> gnu.org, 47260 <at> debbugs.gnu.org, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Mon, 22 Mar 2021 13:58:10 -0400

Ben Sturmfels via Bug reports for GNU Guix writes:

> On Fri, 19 Mar 2021, jgart wrote:
>
>> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>>
>>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>>> improve the no-bundled-JavaScript video/audio playing experience.
>>
>> What are your thoughts on rewriting the jquery? 
>>
>> Should MediaGoblin be using vanilla javascript instead?
>>
>> Some other possibilities could be purescript (https://www.purescript.org) or
>> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
>> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
>> (https://www.mint-lang.com/blog/mint-0.8.0).
>
> Many of the functions we used to use jQuery for are now built into most
> browsers from the last 10 years.
>
> By far the most common are element selectors like:
>
>   var panel = $('#header-panel')
>   var arrow = $('.arrow')
>
> which we just change to:
>
>   var panel = document.querySelector('#header-panel')
>   var arrow = document.querySelector('.arrow')
>
> This is all vanilla JavaScript in individual files imported via <script>
> without any sort of compilation process. This seems to be the simplest
> path to get into distributions like Guix and Debian. The main problem
> with purescript and others is that they require more dependencies, build
> steps and tooling and are known by less people.
>
> We have a few dependencies which will be harder to deal with such as the
> video, audio and 3D model viewers. In the first instance, I think the
> answer will be to do without this JavaScript entirely. That should be
> feasible for video and audio given modern browser support. Perhaps we
> don't ship 3D model support out of the box in distros for now.
>
> Regards,
> Ben

The 3d model support isn't MediaGoblin's most heavily used plugin.  Not
shipping support for that would be fine, I'd think.

 - Chris

Message #26 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: "Dr. Arne Babenhauserheide" <arne_bab <at> web.de>
Cc: 47260 <at> debbugs.gnu.org
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Tue, 30 Mar 2021 15:02:07 +1100

On Mon, 22 Mar 2021, Dr. Arne Babenhauserheide wrote:

> If you need support for m3u-playlists, you can use the player I wrote
> here: https://www.draketo.de/software/m3u-player
> → https://www.draketo.de/software/m3u-player.js (save as utf-8)
> (that m3u-playlists aren’t supported out of the box in most players is a
> strange oversight, the code adds it for video- and audio-tags, License:
> GPLv2 or later — just ask me if you need something else)
>
> There’s also an enhanced version for Freenet, but that has lots of
> performance-changes to work over high-latency networks and with paranoid
> CSP-settings:
> https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802

Thanks Arne! I've forwarded this on to mediagoblin-devel <at> gnu.org so we
don't lose track of it.

Regards,
Ben

Message #29 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org,
 Léo Le Bouter <lle-bout <at> zaclys.net>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Tue, 30 Mar 2021 15:12:48 +1100

Just an update:

On Fri, 19 Mar 2021, Ben Sturmfels wrote:

> 1. Add OGG support to libsndfile which is needed to package
> python-soundfile [patch 47210]

lfam pointed out that this has already been done in 46067, which is now
in core-updates awaiting merging into master:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46067

For the time being, we've also copied the updated libsndfile into
MediaGoblin's guix-env.scm.

> 2. Package python-soundfile (see above). After this the test suite
> should pass 100% with pytest installed from PyPI [patch 47181]

We now have a local copy of python-soundfile in MediaGoblin's
guix-env.scm which is passing all audio tests. Yay!

> 3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
> in Guix seem to be incompatible. After this our test suite should run
> 100% with only dependencies from Guix!

Discovered we'll also need to upgrade Guix's python-wtforms, but in the
mean time, installing only wtforms, pytest, pytest-xdist, pytest-forked
from PyPI allows us to pass the test suite 100% Getting closer!

> ...
> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
> implicated in many bugs anyway, so there may benefits to the project to
> doing this anyway.

I learnt that Celery has a Redis backend, so maybe we don't need to
rewrite just yet.

Regards,
Ben

Message #32 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: "Dr. Arne Babenhauserheide" <arne_bab <at> web.de>
To: Ben Sturmfels <ben <at> sturm.com.au>
Cc: 47260 <at> debbugs.gnu.org
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Tue, 30 Mar 2021 08:40:44 +0200

[Message part 1 (text/plain, inline)]

Ben Sturmfels <ben <at> sturm.com.au> writes:

> On Mon, 22 Mar 2021, Dr. Arne Babenhauserheide wrote:
>
>> If you need support for m3u-playlists, you can use the player I wrote
>> here: https://www.draketo.de/software/m3u-player
>> → https://www.draketo.de/software/m3u-player.js (save as utf-8)
>> (that m3u-playlists aren’t supported out of the box in most players is a
>> strange oversight, the code adds it for video- and audio-tags, License:
>> GPLv2 or later — just ask me if you need something else)
>>
>> There’s also an enhanced version for Freenet, but that has lots of
>> performance-changes to work over high-latency networks and with paranoid
>> CSP-settings:
>> https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802
>
> Thanks Arne! I've forwarded this on to mediagoblin-devel <at> gnu.org so we
> don't lose track of it.

Thank you!

I added one change last week to support mobile browsers which answer
"maybe" to the query `mediaTag.canPlayType('audio/x-mpegurl')` (yes,
seriously, and it is in the spec :-) ).

Also I backported the not freenet specific changes:
- prefetch the next three tracks as blob and keep at most 10 tracks
  cached to allow for fast track skipping (and now actually release the
  memory)
- adjustments to allow for inlining and survive the non-utf8-encoding.
- continue automatically when fetch succeeded if playback was stopped
  because it reached the end (but not if paused).
- minimal mouseover for the back and forward arrows.

When a https-m3u-list refers to a http-file, it falls back from fetching
blobs to rewriting the src-part of the tag (because blobs cannot be
fetched from a less secure resource).

This is how it looks: https://www.draketo.de/software/m3u-player.html

The changes are included in https://www.draketo.de/software/m3u-player.js

You can use it like this:

<script src="m3u-player.js" defer="defer"></script>
<audio src="m3u-player-example-playlist.m3u" controls="controls">
not supported?
</audio>


To make this bug-report independent of my site, here’s the full code:



// [[file:m3u-player.org::*The script][The script:1]]
// @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2-or-Later
const nodes = document.querySelectorAll("audio,video");
const playlists = {};
const prefetchedTracks = new Map(); // use a map for insertion order, so we can just blow away old entries.
// maximum prefetched blobs that are kept.
const MAX_PREFETCH_KEEP = 10;
// maximum allowed number of entries in a playlist to prevent OOM attacks against the browser with self-referencing playlists
const MAX_PLAYLIST_LENGTH = 1000;
const PLAYLIST_MIME_TYPES = ["audio/x-mpegurl", "audio/mpegurl", "application/vnd.apple.mpegurl","application/mpegurl","application/x-mpegurl"];
function stripUrlParameters(link) {
  const url = new URL(link, window.location);
  url.search = "";
  url.hash = "";
  return url.href;
}
function isPlaylist(link) {
  const linkHref = stripUrlParameters(link);
  return linkHref.endsWith(".m3u") || linkHref.endsWith(".m3u8");
}
function isBlob(link) {
  return new URL(link, window.location).protocol == 'blob';
}
function parsePlaylist(textContent) {
  return textContent.match(/^(?!#)(?!\s).*$/mg)
    .filter(s => s); // filter removes empty strings
}
/**
 * Download the given playlist, parse it, and store the tracks in the
 * global playlists object using the url as key.
 *
 * Runs callback once the playlist downloaded successfully.
 */
function fetchPlaylist(url, onload, onerror) {
  const playlistFetcher = new XMLHttpRequest();
  playlistFetcher.open("GET", url, true);
  playlistFetcher.responseType = "blob"; // to get a mime type
  playlistFetcher.onload = () => {
    if (PLAYLIST_MIME_TYPES.includes(playlistFetcher.response.type)) { // security check to ensure that filters have run
      const reader = new FileReader();
      const load = onload; // propagate to inner scope
      reader.addEventListener("loadend", e => {
        playlists[url] = parsePlaylist(reader.result);
        onload();
      });
      reader.readAsText(playlistFetcher.response);
    } else {
      console.error("playlist must have one of the playlist MIME type '" + PLAYLIST_MIME_TYPES + "' but it had MIME type '" + playlistFetcher.response.type + "'.");
      onerror();
    }
  };
  playlistFetcher.onerror = onerror;
  playlistFetcher.abort = onerror;
  playlistFetcher.send();
}
function prefetchTrack(url, onload) {
  if (prefetchedTracks.has(url)) {
    return;
  }
  // first cleanup: kill the oldest entries until we're back at the allowed size
  while (prefetchedTracks.size > MAX_PREFETCH_KEEP) {
    const key = prefetchedTracks.keys().next().value;
    const track = prefetchedTracks.get(key);
    prefetchedTracks.delete(key);
  }
  // first set the prefetched to the url so we will never request twice
  prefetchedTracks.set(url, url);
  // now start replacing it with a blob
  const xhr = new XMLHttpRequest();
  xhr.open("GET", url, true);
  xhr.responseType = "blob";
  xhr.onload = () => {
    prefetchedTracks.set(url, xhr.response);
    if (onload) {
      onload();
    }
  };
  xhr.send();
}
function updateSrc(mediaTag, callback) {
  const playlistUrl = mediaTag.getAttribute("playlist");
  const trackIndex =  mediaTag.getAttribute("track-index");
  // deepcopy playlists to avoid shared mutation
  let playlist = [...playlists[playlistUrl]];
  let trackUrl = playlist[trackIndex];
  // download and splice in playlists as needed
  if (isPlaylist(trackUrl)) {
    if (playlist.length >= MAX_PLAYLIST_LENGTH) {
      // skip playlist if we already have too many tracks
      changeTrack(mediaTag, +1);
    } else {
      // do not use the cached playlist here, though it is tempting: it might genuinely change to allow for updates
      fetchPlaylist(
        trackUrl,
        () => {
          playlist.splice(trackIndex, 1, ...playlists[trackUrl]);
          playlists[playlistUrl] = playlist;
          updateSrc(mediaTag, callback);
        },
        () => callback());
    }
  } else {
    let url = prefetchedTracks.has(trackUrl)
        ? prefetchedTracks.get(trackUrl) instanceof Blob
        ? URL.createObjectURL(prefetchedTracks.get(trackUrl))
        : trackUrl : trackUrl;
    const oldUrl = mediaTag.getAttribute("src");
    mediaTag.setAttribute("src", url);
    // replace the url when done, because a blob from an xhr request
    // is more reliable in the media tag;
    // the normal URL caused jumping prematurely to the next track.
    if (url == trackUrl) {
      prefetchTrack(trackUrl, () => {
        if (mediaTag.paused) {
          if (url == mediaTag.getAttribute("src")) {
            if (mediaTag.currentTime === 0) {
              mediaTag.setAttribute("src", URL.createObjectURL(
                prefetchedTracks.get(url)));
            }
          }
        }
      });
    }
    // allow releasing memory
    if (isBlob(oldUrl)) {
      URL.revokeObjectURL(oldUrl);
    }
    // update title
    mediaTag.parentElement.querySelector(".m3u-player--title").title = trackUrl;
    mediaTag.parentElement.querySelector(".m3u-player--title").textContent = trackUrl;
    // start prefetching the next three tracks.
    for (const i of [1, 2, 3]) {
      if (playlist.length > Number(trackIndex) + i) {
        prefetchTrack(playlist[Number(trackIndex) + i]);
      }
    }
    callback();
  }
}
function changeTrack(mediaTag, diff) {
  const currentTrackIndex = Number(mediaTag.getAttribute("track-index"));
  const nextTrackIndex = currentTrackIndex + diff;
  const tracks = playlists[mediaTag.getAttribute("playlist")];
  if (nextTrackIndex >= 0) { // do not collapse the if clauses with double-and, that does not survive inlining
    if (tracks.length > nextTrackIndex) {
    mediaTag.setAttribute("track-index", nextTrackIndex);
      updateSrc(mediaTag, () => mediaTag.play());
    }
  }
}

/**
 * Turn a media tag into playlist player.
 */
function initPlayer(mediaTag) {
  mediaTag.setAttribute("playlist", mediaTag.getAttribute("src"));
  mediaTag.setAttribute("track-index", 0);
  const url = mediaTag.getAttribute("playlist");
  const wrapper = mediaTag.parentElement.insertBefore(document.createElement("div"), mediaTag);
  const controls = document.createElement("div");
  const left = document.createElement("span");
  const title = document.createElement("span");
  const right = document.createElement("span");
  controls.appendChild(left);
  controls.appendChild(title);
  controls.appendChild(right);
  left.classList.add("m3u-player--left");
  right.classList.add("m3u-player--right");
  title.classList.add("m3u-player--title");
  title.style.overflow = "hidden";
  title.style.textOverflow = "ellipsis";
  title.style.whiteSpace = "nowrap";
  title.style.opacity = "0.3";
  title.style.direction = "rtl"; // for truncation on the left
  title.style.paddingLeft = "0.5em";
  title.style.paddingRight = "0.5em";
  controls.style.display = "flex";
  controls.style.justifyContent = "space-between";
  const styleTag = document.createElement("style");
  styleTag.innerHTML = ".m3u-player--left:hover, .m3u-player--right:hover {color: wheat; background-color: DarkSlateGray}";
  wrapper.appendChild(styleTag);
  wrapper.appendChild(controls);
  controls.style.width = mediaTag.getBoundingClientRect().width.toString() + "px";
  // appending the media tag to the wrapper removes it from the outer scope but keeps the event listeners
  wrapper.appendChild(mediaTag);
  left.innerHTML = "&lt;"; // not textContent, because we MUST escape
                           // the tag here and textContent shows the
                           // escaped version
  left.onclick = () => changeTrack(mediaTag, -1);
  right.innerHTML = "&gt;";
  right.onclick = () => changeTrack(mediaTag, +1);
  fetchPlaylist(
    url,
    () => {
      updateSrc(mediaTag, () => null);
      mediaTag.addEventListener("ended", event => {
        if (mediaTag.currentTime >= mediaTag.duration) {
          changeTrack(mediaTag, +1);
        }
      });
    },
    () => null);
  // keep the controls aligned to the media tag
  mediaTag.resizeObserver = new ResizeObserver(entries => {
    controls.style.width = entries[0].contentRect.width.toString() + "px";
  });
  mediaTag.resizeObserver.observe(mediaTag);
}
function processTag(mediaTag) {
  const canPlayClaim = mediaTag.canPlayType('audio/x-mpegurl');
  let supportsPlaylists = !!canPlayClaim;
  if (canPlayClaim == 'maybe') { // yes, seriously: specced as you only know when you try
    supportsPlaylists = false;
  }
  if (!supportsPlaylists) {
    if (isPlaylist(mediaTag.getAttribute("src"))) {
      initPlayer(mediaTag);
    }
  }
}
document.addEventListener('DOMContentLoaded', () => {
  const nodes = document.querySelectorAll("audio,video");
  nodes.forEach(processTag);
});
// @license-end
// The script:1 ends here




Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[signature.asc (application/pgp-signature, inline)]

Message #35 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org,
 Léo Le Bouter <lle-bout <at> zaclys.net>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Tue, 30 Mar 2021 23:13:06 +1100

On Tue, 30 Mar 2021, Ben Sturmfels wrote:

>> 3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
>> in Guix seem to be incompatible. After this our test suite should run
>> 100% with only dependencies from Guix!
>
> Discovered we'll also need to upgrade Guix's python-wtforms, but in the
> mean time, installing only wtforms, pytest, pytest-xdist, pytest-forked
> from PyPI allows us to pass the test suite 100% Getting closer!

Pytest issues have now been resolved. We now need ZERO packages from
PyPI for a basic install and full test suite run. :)

Regards,
Ben

Message #38 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org,
 Léo Le Bouter <lle-bout <at> zaclys.net>, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Thu, 01 Apr 2021 13:03:08 +1100

On Tue, 30 Mar 2021, Ben Sturmfels wrote:

> On Fri, 19 Mar 2021, Ben Sturmfels wrote:

>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>> implicated in many bugs anyway, so there may benefits to the project to
>> doing this anyway.
>
> I learnt that Celery has a Redis backend, so maybe we don't need to
> rewrite just yet.

It turns out that MediaGoblin's Celery-based media processing backend
work out of the box by simply configuring:

  [celery]
  BROKER_URL = "redis://"

(There seems to be an unrelated bug where media is marked as failed after
restarting Celery, possibly tied to sqlite. We've had reports of this
with a RabbitMQ broker too though.)


This means our shorter to-do list is now:

1. Upstream our new python-soundfile Guix package from guix-env.scm when
core-updates is merged.

2. Upstream our upgraded python-wtforms package.

6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
Video and audio are essentially functional without the NPM installed
players. Some later refinements perhaps.

4. Package MediaGoblin itself. The build process is ./configure/make
which is a bit weird for a Python project.

5. Get a basic Guix service working, with sqlite3 and without the
offloaded media transcoding currently using Celery task queue with a
Redis broker.

7. Work out why H264 support is missing.

8. Figure out how to deal with translations.

9. Add a PostgreSQL database to the Guix service instead of sqlite3.

Regards,
Ben

Message #41 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org,
 Léo Le Bouter <lle-bout <at> zaclys.net>, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Tue, 06 Apr 2021 00:17:11 +1000

On Thu, 01 Apr 2021, Ben Sturmfels wrote:

> 7. Work out why H264 support is missing.

This is now fixed MediaGoblin's master branch guix-env.scm by adding
gst-libav to propagated inputs.

Message #44 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Léo Le Bouter <lle-bout <at> zaclys.net>
To: Ben Sturmfels <ben <at> sturm.com.au>, 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Mon, 05 Apr 2021 17:50:40 +0200

[Message part 1 (text/plain, inline)]

On Tue, 2021-04-06 at 00:17 +1000, Ben Sturmfels wrote:
> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
> 
> > 7. Work out why H264 support is missing.
> 
> This is now fixed MediaGoblin's master branch guix-env.scm by adding
> gst-libav to propagated inputs.

Hello!

I suggest not using propagated-inputs because they are likely to cause
conflicts in profiles when used. I suggest intead creating wrapper
scripts that append to the PATH variable.

Examples of such wrapper scripts can be found:

- 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=373e5fc96724fd38bb1263e4af90932ea36f596b
(PYTHONPATH more so)
- 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874
(Append xdg-utils to PATH)

Let me know if there's any questions about creating such wrappers.

Léo 

[signature.asc (application/pgp-signature, inline)]

Message #47 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org,
 Léo Le Bouter <lle-bout <at> zaclys.net>, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Tue, 06 Apr 2021 22:01:56 +1000

[Message part 1 (text/plain, inline)]

On Thu, 01 Apr 2021, Ben Sturmfels wrote:

> 5. Get a basic Guix service working, with sqlite3 and without the
> offloaded media transcoding currently using Celery task queue with a
> Redis broker.

Woo! After a lot of trial and error, I finally have a basic MediaGoblin
running entirely under Guix with no virtualenv trickery!

After applying the attached patch to my guix repo, I grab a copy of the
basic config files and enable audio and video:

  curl https://git.savannah.gnu.org/cgit/mediagoblin.git/plain/mediagoblin.example.ini > mediagoblin.ini
  curl https://git.savannah.gnu.org/cgit/mediagoblin.git/plain/paste.ini > paste.ini
  echo "[[mediagoblin.media_types.audio]]" >> mediagoblin.ini
  echo "[[mediagoblin.media_types.video]]" >> mediagoblin.ini

Build MediaGoblin, which downloads from our master branch and runs the
full test suite successfully:

  ~/ws/guix/pre-inst-env guix build mediagoblin

Then install MediaGoblin in a container (not working in a non-container
guix environment or without explicit "python"):

  ~/ws/guix/pre-inst-env guix environment --container --network --share=$HOME/.bash_history --ad-hoc mediagoblin python

Create an sqlite3 database and add a user:

  gmg dbupdate
  gmg adduser --username admin --password a --email admin <at> example.com
  gmg changepw admin a

Upload an image, audio and video via CLI:

  gmg addmedia admin image.jpg
  gmg addmedia admin audio.wav
  gmg addmedia admin video.mp4

Start the web interface:

  CELERY_ALWAYS_EAGER=true paster serve paste.ini

The web interface is working. Looks like we're missing some CSS
(probably due to files not being included in the setuptools package),
but that's a minor issue.

Getting there!

Regards,
Ben

[0001-Add-MediaGoblin-package.patch (text/x-patch, attachment)]

Message #50 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: Léo Le Bouter <lle-bout <at> zaclys.net>
Cc: cwebber <at> dustycloud.org, 47260 <at> debbugs.gnu.org, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Tue, 06 Apr 2021 22:05:19 +1000

On Mon, 05 Apr 2021, Léo Le Bouter wrote:

> On Tue, 2021-04-06 at 00:17 +1000, Ben Sturmfels wrote:
>> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
>> 
>> > 7. Work out why H264 support is missing.
>> 
>> This is now fixed MediaGoblin's master branch guix-env.scm by adding
>> gst-libav to propagated inputs.
>
> Hello!
>
> I suggest not using propagated-inputs because they are likely to cause
> conflicts in profiles when used. I suggest intead creating wrapper
> scripts that append to the PATH variable.
>
> Examples of such wrapper scripts can be found:
>
> - 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=373e5fc96724fd38bb1263e4af90932ea36f596b
> (PYTHONPATH more so)
> - 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874
> (Append xdg-utils to PATH)
>
> Let me know if there's any questions about creating such wrappers.

Thanks Léo, I'll look into this!

Regards,
Ben

Message #53 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org,
 Léo Le Bouter <lle-bout <at> zaclys.net>, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Wed, 07 Apr 2021 23:15:57 +1000

On Tue, 06 Apr 2021, Ben Sturmfels wrote:

> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
>
>> 5. Get a basic Guix service working, with sqlite3 and without the
>> offloaded media transcoding currently using Celery task queue with a
>> Redis broker.
>
> Woo! After a lot of trial and error, I finally have a basic MediaGoblin
> running entirely under Guix with no virtualenv trickery!
>
> After applying the attached patch to my guix repo...

Even simpler, I've now created a Guix channel for our work-in-progress
MediaGoblin package/service. See the README for instructions on enabling
the channel and installing MediaGoblin:

https://gitlab.com/BenSturmfels/mediagoblin-guix/-/blob/master/README.md

Once the channel is enabled, you can `guix install mediagoblin` and run
`gmg` commands and the web interface. All documented in the above README.

Regards,
Ben

Message #56 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Dr. Arne Babenhauserheide" <arne_bab <at> web.de>
To: Ben Sturmfels <ben <at> sturm.com.au>
Cc: cwebber <at> dustycloud.org, bug-guix <at> gnu.org, 47260 <at> debbugs.gnu.org,
 jgart <jgart <at> dismail.de>, Léo Le Bouter <lle-bout <at> zaclys.net>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Tue, 04 May 2021 22:58:00 +0200

[Message part 1 (text/plain, inline)]

Hi,

I just added non-flickering video-change to the m3u-player. Attaching
the file. I thought that could be useful for MediaGoblin. The file is
attached.

[m3u-player.js (application/octet-stream, attachment)]

[Message part 3 (text/plain, inline)]

Best wishes,
Arne


Ben Sturmfels via Bug reports for GNU Guix <bug-guix <at> gnu.org> writes:

> On Tue, 30 Mar 2021, Ben Sturmfels wrote:
>
>> On Fri, 19 Mar 2021, Ben Sturmfels wrote:
>
>>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>>> implicated in many bugs anyway, so there may benefits to the project to
>>> doing this anyway.
>>
>> I learnt that Celery has a Redis backend, so maybe we don't need to
>> rewrite just yet.
>
> It turns out that MediaGoblin's Celery-based media processing backend
> work out of the box by simply configuring:
>
>   [celery]
>   BROKER_URL = "redis://"
>
> (There seems to be an unrelated bug where media is marked as failed after
> restarting Celery, possibly tied to sqlite. We've had reports of this
> with a RabbitMQ broker too though.)
>
>
> This means our shorter to-do list is now:
>
> 1. Upstream our new python-soundfile Guix package from guix-env.scm when
> core-updates is merged.
>
> 2. Upstream our upgraded python-wtforms package.
>
> 6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
> Video and audio are essentially functional without the NPM installed
> players. Some later refinements perhaps.
>
> 4. Package MediaGoblin itself. The build process is ./configure/make
> which is a bit weird for a Python project.
>
> 5. Get a basic Guix service working, with sqlite3 and without the
> offloaded media transcoding currently using Celery task queue with a
> Redis broker.
>
> 7. Work out why H264 support is missing.
>
> 8. Figure out how to deal with translations.
>
> 9. Add a PostgreSQL database to the Guix service instead of sqlite3.
>
> Regards,
> Ben


-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[signature.asc (application/pgp-signature, inline)]

Message #62 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: "Dr. Arne Babenhauserheide" <arne_bab <at> web.de>
Cc: cwebber <at> dustycloud.org, bug-guix <at> gnu.org, 47260 <at> debbugs.gnu.org,
 jgart <jgart <at> dismail.de>,
 Léo Le Bouter <lle-bout <at> zaclys.net>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Thu, 06 May 2021 10:49:38 +1000

Thanks for the update Arne. This issue is specifically about Guix
packaging, so to save us losing track of your update, please post it to
the dedicated mediagoblin-devel <at> gnu.org thread we started a couple of
months back:

https://lists.gnu.org/archive/html/mediagoblin-devel/2021-03/msg00026.html

Thanks again,
Ben

On Wed, 05 May 2021, Arne Babenhauserheide wrote:

> Hi,
>
> I just added non-flickering video-change to the m3u-player. Attaching
> the file. I thought that could be useful for MediaGoblin. The file is
> attached.
>
>
> Best wishes,
> Arne
>
>
> Ben Sturmfels via Bug reports for GNU Guix <bug-guix <at> gnu.org> writes:
>
>> On Tue, 30 Mar 2021, Ben Sturmfels wrote:
>>
>>> On Fri, 19 Mar 2021, Ben Sturmfels wrote:
>>
>>>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>>>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>>>> implicated in many bugs anyway, so there may benefits to the project to
>>>> doing this anyway.
>>>
>>> I learnt that Celery has a Redis backend, so maybe we don't need to
>>> rewrite just yet.
>>
>> It turns out that MediaGoblin's Celery-based media processing backend
>> work out of the box by simply configuring:
>>
>>   [celery]
>>   BROKER_URL = "redis://"
>>
>> (There seems to be an unrelated bug where media is marked as failed after
>> restarting Celery, possibly tied to sqlite. We've had reports of this
>> with a RabbitMQ broker too though.)
>>
>>
>> This means our shorter to-do list is now:
>>
>> 1. Upstream our new python-soundfile Guix package from guix-env.scm when
>> core-updates is merged.
>>
>> 2. Upstream our upgraded python-wtforms package.
>>
>> 6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
>> Video and audio are essentially functional without the NPM installed
>> players. Some later refinements perhaps.
>>
>> 4. Package MediaGoblin itself. The build process is ./configure/make
>> which is a bit weird for a Python project.
>>
>> 5. Get a basic Guix service working, with sqlite3 and without the
>> offloaded media transcoding currently using Celery task queue with a
>> Redis broker.
>>
>> 7. Work out why H264 support is missing.
>>
>> 8. Figure out how to deal with translations.
>>
>> 9. Add a PostgreSQL database to the Guix service instead of sqlite3.
>>
>> Regards,
>> Ben

Message #68 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org, Raghav Gururajan <rg <at> raghavgururajan.name>,
 Léo Le Bouter <lle-bout <at> zaclys.net>, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Sun, 12 Sep 2021 12:38:00 +1000

Just a progress update - Raghav and Jgart have now packaged and updated
python-wtforms, thank you both!

All that remains now for a passing test suite (and OGG Vorbis support)
is the modified libsndfile that's waiting in Guix's core-updates branch.
Very close now.

Regards,
Ben

Message #71 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org, Raghav Gururajan <rg <at> raghavgururajan.name>,
 Léo Le Bouter <lle-bout <at> zaclys.net>, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Mon, 13 Sep 2021 14:06:03 +1000

I've now written up all the progress in our MediaGoblin Guix channel README:

https://git.sr.ht/~mediagoblin/mediagoblin-guix/tree/master/item/README.md

In short MediaGoblin can be installed as a Guix package with no external
dependencies or Python virtualenvs. After some slightly clumsy static
files configuration the web interface runs successfully based on an
SQLite database. The Celery backend run successfully. Images and video
can be uploaded and viewed successfully. Video plays via the stock
browser player but doesn't allow selection of video quality. Audio is
broken pending merge of updated libsndfile from Guix's "core-updates"
branch.

To try it out, follow the instructions in "Install via load-path",
followed by "Run MediaGoblin" from the README mentioned above.

Regards,
Ben

Message #74 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: 47260 <at> debbugs.gnu.org
Cc: cwebber <at> dustycloud.org, Raghav Gururajan <rg <at> raghavgururajan.name>,
 Léo Le Bouter <lle-bout <at> zaclys.net>, jgart <jgart <at> dismail.de>
Subject: Re: bug#47260: Package GNU MediaGoblin as a Guix service
Date: Sat, 18 Sep 2021 00:20:32 +1000

On Mon, 13 Sep 2021, Ben Sturmfels wrote:

I've now updated MediaGoblin to find static assets relative to the code,
and hacked in two new entrypoints `gmg serve` and `gmg celery`. Together
these changes mean that we can now mostly use `inputs` rather than
`propagated-inputs` and that the CSS and images are showing up when
installed as a user package (not service)

Here's the updated channel:

https://git.sr.ht/~mediagoblin/mediagoblin-guix/tree/master/item/README.md

I think we're close to being able to propose a patch for a MediaGoblin
package that can be run standalone. Haven't yet worked on a service.

Message #77 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: jgart <jgart <at> dismail.de>
To: mediagoblin-devel <at> gnu.org
Cc: 47260 <at> debbugs.gnu.org
Subject: Wrapping binaries in MediaGoblin Guix Package
Date: Tue, 5 Oct 2021 00:34:20 -0400

Just a note that we'll probably have to wrap the binaries that are in this 
module when we finish making the guix package:

mediagoblin/media_types/pdf/processing.py

I see that pdftocairo, pdfinfo, and unoconv probably need to be wrapped to point
to the executable that is in /gnu/store.

See this issue for a similar discussion about wrapping binaries with guix:

https://issues.guix.gnu.org/50833

Message #80 received at 47260 <at> debbugs.gnu.org (full text, mbox):

From: Ben Sturmfels <ben <at> sturm.com.au>
To: jgart <jgart <at> dismail.de>
Cc: 47260 <at> debbugs.gnu.org, mediagoblin-devel <at> gnu.org
Subject: Re: Wrapping binaries in MediaGoblin Guix Package
Date: Tue, 05 Oct 2021 16:34:03 +1100

On Tue, 05 Oct 2021, jgart wrote:

> Just a note that we'll probably have to wrap the binaries that are in this 
> module when we finish making the guix package:
>
> mediagoblin/media_types/pdf/processing.py
>
> I see that pdftocairo, pdfinfo, and unoconv probably need to be wrapped to point
> to the executable that is in /gnu/store.
>
> See this issue for a similar discussion about wrapping binaries with guix:
>
> https://issues.guix.gnu.org/50833

Thanks, I've made a note in guix-env.scm to remind me when we look at
PDF support in Guix.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.