• soupgate and alternatives

    From Oli@21:3/102 to All on Wednesday, January 27, 2021 19:13:04
    Soupgate looked for 0000????.msg Soup files and couldn't open the 0000????.MSG Soup files. This problem was introduced when I changed the extension for Fidonet messages from ".MSG" to ".msg". It exposed a hidden bug, because Soupgate used the Fidonet .msg extension for Soup files instead of the Soup extension which is still defined as ".MSG".

    Unfortunately the more I use soupgate the less I like it. It's simple to use, but it's approach to FTN<->Internet message gating is way to simplistic. No real charset conversions, no TZUTC support, dumping the complete RFC822 header into kludge lines (even multi-line fields), non-Gatebau compliant MSGID/Message-IDs. It's the worst case scenario for dupes: changed datetime, changed text body, changed MGSID, a bunch of weird kludges.

    I wonder if it is best to look for better alternatives. I guess even a rewrite from scratch would be not much more work then extending and bugfixing Soupgate and Spoon.

    I downloaded some of the Gatebau '94 and '97 documentation. Mostly in German, but the most important one is available in English too:
    https://fidogate.sourceforge.net/msgid.html

    Soupgate does it all wrong ;).

    I don't mind doing a bit more bug fixing on Soupgate, but I don't think I want to dive into the "double free" Spoon code. I let Spoon connect to my JamNNTPd server. It didn't even try to login a user (username and password is in the config) and crashed the server and then itself. Isn't it possible to run vSoup in wine? Or is there some other Soup client that can be used on linux?

    Btw, I just stumbled upon a Soupgate thread on LINUX_BBS from last year (30 Jun 20 "Soupgate / Spoon"). Same problems.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Thursday, January 28, 2021 16:39:03
    On 27 Jan 2021 at 07:13p, Oli pondered and said...

    Soupgate looked for 0000????.msg Soup files and couldn't open the 0000????.MSG Soup files. This problem was introduced when I changed the extension for Fidonet messages from ".MSG" to ".msg". It exposed a

    Good catch... I also found that with an earlier version when running the
    export function Soupgate was looking for .CLO not .clo files...
    Perhaps this is the same issue?

    Unfortunately the more I use soupgate the less I like it. It's simple to use, but it's approach to FTN<->Internet message gating is way to simplistic. No real charset conversions, no TZUTC support, dumping the complete RFC822 header into kludge lines (even multi-line fields), non-Gatebau compliant MSGID/Message-IDs. It's the worst case scenario
    for dupes: changed datetime, changed text body, changed MGSID, a bunch
    of weird kludges.

    Many kludges can be disabled by config file. I agree it's not perfect but the windows version has been working OK for several years for me without too many niggles on either side of the gate.

    I wonder if it is best to look for better alternatives. I guess even a rewrite from scratch would be not much more work then extending and bugfixing Soupgate and Spoon.

    One reason I like the combo is that they are two stand alone exec files that
    I can install, config and run. It doesn't require a huge overhead to install something more complex.. But I agree fixing bugs and extending features would be great. Just wish I had the coding skills to do it :(

    I don't mind doing a bit more bug fixing on Soupgate, but I don't think
    I want to dive into the "double free" Spoon code. I let Spoon connect to my JamNNTPd server. It didn't even try to login a user (username and password is in the config) and crashed the server and then itself. Isn't it possible to run vSoup in wine? Or is there some other Soup client
    that can be used on linux?

    It may be the way spoon talks to your NNTP is different to the INN NNTP I am using. I know the later compiled version we have got working to pull down
    news seems to work alright and I can post to NNTP which does require a
    username and password on my system.. and that works also... it's just the
    hand off at the end when the system borks on the known error that is an issue

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Thursday, January 28, 2021 14:42:04
    Avon wrote (2021-01-28):

    On 27 Jan 2021 at 07:13p, Oli pondered and said...

    Soupgate looked for 0000????.msg Soup files and couldn't open the
    0000????.MSG Soup files. This problem was introduced when I changed
    the extension for Fidonet messages from ".MSG" to ".msg". It
    exposed a

    Good catch... I also found that with an earlier version when running the export function Soupgate was looking for .CLO not .clo files...
    Perhaps this is the same issue?

    Yes and no. It's defined in soupgate.c (and can be changed easily):

    const char binkmailext[BINK_ALLTYPES][5]
    = {".out",".cut",".dut",".hut"};

    const char binkflowext[BINK_ALLTYPES][5]
    = {".flo",".clo",".dlo",".hlo"};

    const char packetext[5]
    = ".pkt";

    const char messageext[5]
    = ".msg";

    const char soupmsgext[5]
    = ".MSG";

    The problem with "soupgate import" was that it used messageext, which is the extension for ArcMail (netmail) messages. The extension for the Soup files is defined by soupmsgext, but never was used on import. As long as both were ".MSG" in uppercase it didn't make a difference, but when I changed messageext from ".MSG" to ".msg" it didn't work as expected anymore.

    Now it's fixed and they should work independent from each other.

    Unfortunately the more I use soupgate the less I like it. It's
    simple to use, but it's approach to FTN<->Internet message gating
    is way to simplistic. No real charset conversions, no TZUTC
    support, dumping the complete RFC822 header into kludge lines (even
    multi-line fields), non-Gatebau compliant MSGID/Message-IDs. It's
    the worst case scenario for dupes: changed datetime, changed text
    body, changed MGSID, a bunch of weird kludges.

    Many kludges can be disabled by config file. I agree it's not perfect but the windows version has been working OK for several years for me without too many niggles on either side of the gate.

    There are standards and best practices for a reason. It might work fine for you now, but in another scenario it might generate all kinds of weird side-effects and problems. Like the Mystic tosser with all it's non-standard behavior worked okay, until it didn't (AFAIK the various bugs have been fixed).

    Soupgate is creating a real mess at re-importing and translating MSGID/REPLYs. One problem is that even for local use reply linking cannot work, when MSGID / Message-IDs aren't translated transparently back and forth.

    Btw, I don't think binkp.net is meant to be used for mail/news gating and that fidonet.org would still be the better (only correct?) option for the Message-ID domain.

    I wonder if it is best to look for better alternatives. I guess
    even a rewrite from scratch would be not much more work then
    extending and bugfixing Soupgate and Spoon.

    One reason I like the combo is that they are two stand alone exec files that I can install, config and run. It doesn't require a huge overhead to install something more complex..

    That's what I liked with soupgate too ...

    But I agree fixing bugs and extending
    features would be great. Just wish I had the coding skills to do it :(

    .... but then it's maybe more work (for us) to get it in a good state than learning to configure a complex monster like fidogate.

    ---
    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Friday, January 29, 2021 10:31:08
    On 28 Jan 2021 at 02:42p, Oli pondered and said...

    changed messageext from ".MSG" to ".msg" it didn't work as expected anymore.

    Now it's fixed and they should work independent from each other.

    Thanks for this explanation and some insights as to how that code works.

    There are standards and best practices for a reason. It might work fine for you now, but in another scenario it might generate all kinds of
    weird side-effects and problems.

    Fair enough.

    Like the Mystic tosser with all it's
    non-standard behavior worked okay, until it didn't (AFAIK the various
    bugs have been fixed).

    I'm not going to get into the pros and cons of Mystic but agree each system
    has it's advantages and disadvantages. Running all of fsxNet with Mystic HUBs was fine but when some were not using the same tossing software it became apparent there were differences in the way packets were being treated.

    Soupgate is creating a real mess at re-importing and translating MSGID/REPLYs. One problem is that even for local use reply linking
    cannot work, when MSGID / Message-IDs aren't translated transparently
    back and forth.

    I think there may be a setting in the ini to adjust some of this?

    Btw, I don't think binkp.net is meant to be used for mail/news gating
    and that fidonet.org would still be the better (only correct?) option
    for the Message-ID domain.

    I'd only used is as I thought the original fidonet domain used that shipped with the initial files had been lost to a domain name squatter. It can be easily adjusted in the ini thankfully.

    One reason I like the combo is that they are two stand alone exec file that I can install, config and run. It doesn't require a huge overhead install something more complex..

    That's what I liked with soupgate too ...

    Simple is good.

    But I agree fixing bugs and extending
    features would be great. Just wish I had the coding skills to do it :(

    .... but then it's maybe more work (for us) to get it in a good state
    than learning to configure a complex monster like fidogate.

    I've not yet had time to rebuild the new code and test it but will look to do so today. In terms of moving 3:770/1 over to HPT I was wanting to get the gating software running first for Liunx but may well just fudge some shuffle
    of packets between Linux and Windows to keep using the software I have
    running under Windows for Soupgate and VSoup while a better more long term solution for gating under Linux can be found.

    Fidogate (I think) needs to be on the same system as an NNTP server (not
    sure) and if so it will be a bit of work for me to setup as I am also wanting to look at migrating my NNTP setup over to the same system that will run the Fido hub... but one thing at a time :)

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Oli@21:3/102 to Avon on Friday, January 29, 2021 12:49:06
    Avon wrote (2021-01-29):

    changed messageext from ".MSG" to ".msg" it didn't work as expected
    anymore.

    Now it's fixed and they should work independent from each other.

    Thanks for this explanation and some insights as to how that code works.

    I found the specification for the "Simple Offline USENET Packet Format (SOUP) Version 1.2". For SOUP files it's always upper case: "Each of the filenames in a packet should be stored in upper case on those systems where case matters (e.g. Unix)". So I don't expect any further problems with filenames.

    Soupgate is creating a real mess at re-importing and translating
    MSGID/REPLYs. One problem is that even for local use reply linking
    cannot work, when MSGID / Message-IDs aren't translated
    transparently back and forth.

    I think there may be a setting in the ini to adjust some of this?

    I don't see a setting that would help. This has to be changed in the code.

    Btw, I don't think binkp.net is meant to be used for mail/news
    gating and that fidonet.org would still be the better (only
    correct?) option for the Message-ID domain.

    I'd only used is as I thought the original fidonet domain used that
    shipped with the initial files had been lost to a domain name squatter.
    It can be easily adjusted in the ini thankfully.

    I think that was fidonet.net not .org and it is only relevant for DNS (binkp mailers calling each other).

    From binkd's HISTORY file:
    2011/08/17 12:02:59 1.0a-560 gul
    Default root-domain fidonet.net changed to binkp.net

    One reason I like the combo is that they are two stand alone
    exec file that I can install, config and run. It doesn't
    require a huge overhead install something more complex..

    That's what I liked with soupgate too ...

    Simple is good.

    I remember that someone complained about hpt being too complex ... (it wasn't me, was it?).

    .... but then it's maybe more work (for us) to get it in a good
    state than learning to configure a complex monster like fidogate.

    I've not yet had time to rebuild the new code and test it but will look
    to do so today.

    If you are lucky, soupgate will work as expected. I thought a bit more about soupgate and maybe it's still a good idea to keep it alive.

    I also checked out Fidogate and instantly lost all motivation to learn it. Maybe I'll try another day.

    Fidogate (I think) needs to be on the same system as an NNTP server (not sure) and if so it will be a bit of work for me to setup as I am also wanting to look at migrating my NNTP setup over to the same system that will run the Fido hub... but one thing at a time :)

    Isn't it possible to send it via rnews to an nntpd server on another machine?

    https://www.eyrie.org/~eagle/software/inn/docs-2.6/rnews.html

    * Origin: . (21:3/102)
  • From Avon@21:1/101 to Oli on Saturday, January 30, 2021 10:10:11
    On 29 Jan 2021 at 12:49p, Oli pondered and said...

    I found the specification for the "Simple Offline USENET Packet Format (SOUP) Version 1.2". For SOUP files it's always upper case: "Each of the filenames in a packet should be stored in upper case on those systems where case matters (e.g. Unix)". So I don't expect any further problems with filenames.

    interesting, thanks for the details... yep I concur.

    I think that was fidonet.net not .org and it is only relevant for DNS (binkp mailers calling each other).

    ah

    I remember that someone complained about hpt being too complex ... (it wasn't me, was it?).

    Hahah, we'll I'm getting better at HPT and soon to throw a switch or two on that software.

    I've not yet had time to rebuild the new code and test it but will loo to do so today.

    If you are lucky, soupgate will work as expected. I thought a bit more about soupgate and maybe it's still a good idea to keep it alive.

    Tests last night seemed to be fine, import and export, it's only Soup and getting stuff posted back out to NNTP successfully that now seems to be the issue.

    --- Mystic BBS v1.12 A46 2020/08/26 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)