Er mogen op dit forum GEEN verzoekjes of requests gevraagd worden, dit staat ook in de regels die je accepteert bij het registreren op dit forum, hiervoor krijg je direct een permanente ban.

How do Spotnet "spots" interact with retention?

This Forum is created for our international residents in The Netherlands.
It is not permitted to link to NZB indexes / copyrighted content in this forum.
We only provide support for technical problems regarding different applications.

Have fun in this English support section!

Moderator: Moderator Team

Forumregels
LEES: Forum Regels!

Topicstarter
OhneHerren
Berichten: 3
Lid geworden op: 21 jan 2016, 06:11
Reputation: 0
Usenet: UsenetBucket
Internet: Australian ISP

How do Spotnet "spots" interact with retention?

#1

Bericht door OhneHerren »

I'm setting up Spotweb on a Linux server and I'm a bit confused about the underlying technology. It's all working and looking great, but I need some more info to set it up properly. Firstly I'm not sure which provider I should use with it. As far as I understand it currently I can only retrieve as many spots as my Usenet provider can retrieve from the relevant groups. At the moment the oldest spots my provider (UsenetBucket) can find are from 5 years and 11 months ago. If I switch the backend of Spotweb to a provider with greater retention (e.g. my block account with NewsGroupDirect) will I find spots from longer ago than that? Additionally how much bandwidth would a regularly updating spotweb system consume? Currently there's about a 700 day difference between the oldest spot I can see with Spotweb (using UsenetBucket) and the oldest article I could theoretically grab with NewsGroupDirects' retention (provided their retention count has increased linearly since their last announcement).

I also notice that there are different groups for both the "header group" and the "nzb group"? How does this work? Are the "headers" the actual spots? And are the nzbs uploaded to the nzb group at the same time as the spot is uploaded to the header group?

If a Usenet Provider with a longer retention time won't allow me to see older spots or if the bandwidth costs are too high on my block account then I'll stick with just grabbing the relatively younger posts. I wish I could answer these questions myself but I'm having trouble finding much English info on how Spotnet works. Thanks for you help.
Neem je online privacy serieus en gebruik een VPN verbinding!

Met een VPN verbinding van IPVanish wordt uw gehele internet verbinding versleuteld

U kunt dus veilig browsen, Torrents downloaden, Usenet downloaden en onderling versleutelde verbindingen opzetten.


Gebruikersavatar

mesa57
Moderator Team Lid
Berichten: 1800
Lid geworden op: 13 feb 2011, 19:03
Topics answered: 2
Reputation: 15

Re: How do Spotnet "spots" interact with retention?

#2

Bericht door mesa57 »

There is a special spotweb forum for spotweb related question, but I cannot move you're message.
Anyway :
- Switching to another provider while the database is filled won't add older spots, as spots are retrieved from the newest and so on. So if you want older spots in you're database you can switch to an other provider, but you have to re-initialise you're database.
- It is correct that headers and nzb's are in different groups. For older spots (2011) there are no nzb's present, you have to search an nzbindexer for the filenames. For newer spots the nzb is uploaded to alt.binaries.ftd at the same time the spot is uploaded to free.pt

But be aware : many provider have a long retention for headers and a lower for binaries. That leads to the problem that for older spots you cannot retrieve the nzb anymore.

As to the headers : these are not the complete spots but just an extraction of information in the subject headers of the spots in free.pt. The complete headers+body is retrieved when you're open the spot.

And of course : feel free to ask any other or additional questions :)
Gebruikersavatar

Blauwbek
Moderator Team Leider
Berichten: 2761
Lid geworden op: 04 mar 2011, 17:29
Reputation: 0
Usenet: UNS
Internet: UPC
Locatie: Blauwbek-PC
Contacteer:

Re: How do Spotnet "spots" interact with retention?

#3

Bericht door Blauwbek »

English support forum is better to ensure better readability for those that don't speak dutch :)
Regels/Rules | pySabRename

Er zijn weer spotnet updates, klik hier!

API's kijken doe je maar in de dierentuin!

Topicstarter
OhneHerren
Berichten: 3
Lid geworden op: 21 jan 2016, 06:11
Reputation: 0
Usenet: UsenetBucket
Internet: Australian ISP

Re: How do Spotnet "spots" interact with retention?

#4

Bericht door OhneHerren »

Thanks very much for getting back to me mesa57.
mesa57 schreef:- It is correct that headers and nzb's are in different groups. For older spots (2011) there are no nzb's present, you have to search an nzbindexer for the filenames. For newer spots the nzb is uploaded to alt.binaries.ftd at the same time the spot is uploaded to free.pt

But be aware : many provider have a long retention for headers and a lower for binaries. That leads to the problem that for older spots you cannot retrieve the nzb anymore.
Okay so you're saying sometimes I will be able to view the spot but not download the nzb. I have experienced this. It is my understanding that content may be "spotted" at any time after it is uploaded. So it would be possible for there to be a spot I am viewing from three years ago but the nzb file attached to that spot refers to content that is older (for example, five years ago). Is this correct?
mesa57 schreef:As to the headers : these are not the complete spots but just an extraction of information in the subject headers of the spots in free.pt. The complete headers+body is retrieved when you're open the spot.
What do you mean by headers+body? Do you mean the headers are the information about the spot (filesize et cetera) and the body provides is where the nzb file is stored? And where are the comments on a spot stored? Perhaps if you could link me to an example of a "header" and "body" post I could understand better. I'm perhaps not likely to use Spotnet extensively due to the mostly Dutch content it catalogues but I am interested in understanding the technology nonetheless. Maybe there is a detailed Dutch article on how Spotnet works which Google translate could make some sense of?

Oh and also, if I leave a Spotnet system running for a year, how much download bandwidth would it consume to stay up to date?

Thanks again for your help. :)
Gebruikersavatar

mesa57
Moderator Team Lid
Berichten: 1800
Lid geworden op: 13 feb 2011, 19:03
Topics answered: 2
Reputation: 15

Re: How do Spotnet "spots" interact with retention?

#5

Bericht door mesa57 »

The NZB can indeed point to files which are even older then the NZB itselfs.
The headers come from the subject of the post and contain all information that is shown in the list.
For details, the whole message had to be read (much slower).
For more technical details you can look at the spotnet source code which is on github. There is also a wiki there explaining to some degree the headers.
Spotnet uses very little bandwidth to load the newest headers as it retrieves only the subject from the messages.
Comments are stored in free.usenet and have a reference to the spot.

Topicstarter
OhneHerren
Berichten: 3
Lid geworden op: 21 jan 2016, 06:11
Reputation: 0
Usenet: UsenetBucket
Internet: Australian ISP

Re: How do Spotnet "spots" interact with retention?

#6

Bericht door OhneHerren »

Ah yes I see Spotweb's database is only 1GB for 5 years of headers.

I think I am coming to some understanding now. That github repo is very useful.

There is indeed a discrepancy between the two Usenet providers I have. The first reports that there is an "Appr. Message count: 2748635" while the second can find "2247994" messages. From this I would expect that the first provider can find older spots than the second, and yet the oldest spots in both cases are the same content. Do you have any idea what this means? Here is an excerpt of the log in both cases.
Plaats reactie Vorig onderwerpVolgend onderwerp
Spot-net.nl Forums : Disclaimer