There are a few important points to be considered regarding the meaning of the words that [they] were to suffer affliction.
The first question is whether Paul is reporting his exact words (direct speech), or just indicating the content of what he had said (indirect speech). Although nearly all the English translations represent this statement as indirect speech, it is likely that Paul is using direct speech: (a) in 2 Thessalonians 3:10 we have a very similar construction which is a clear case of direct speech (as virtually all the English translations recognize); (b) the main verb in this phrase is in the present tense, which is more appropriate for direct speech than indirect speech; (c) a direct quote of his exact words would make Paul’s statement here much more vivid.1,2
We could translate 1 Thessalonians 3:4a as: In fact, when we were with you we kept forewarning you: We are destined to suffer affliction.
4 For when we were with you, we kept telling you beforehand that we were to suffer affliction, just as it has come to pass, and just as you know.