GeekTips

Posting again as I've updated the awk script. Much better now only prints line numbers where the lines are 3 or 4 subtitle lines a way meaning much more likely to be hallucinating or repeating segments.

So found an awk script that does what I just turned into into a one-liner that's all

repeatsubs

for WebVTT subtitles

awk '{ x = lines[$0]["count"]++; lines[$0]["NR"][x] = NR; }END {fmt_s = "%sx %" max "-s %s\n\n"; for (i in lines) {if (lines[i]["count"] > 2 && lines[i]["count"] < 99) {for (j = 0; j < lines[i]["count"]; j++) {s = s lines[i]["NR"][j] ", ";} s = substr(s, 1, length(s) - 2); printf(fmt_s, lines[i]["count"], i, "\n" s ); s = "";}}}' *.vtt | awk -F, '$2 < $1+13' | grep -E -A1 '^[0-9]{,2}x' --color=always | less -r

for SRT subtitles

awk '{ x = lines[$0]["count"]++; lines[$0]["NR"][x] = NR; }END {fmt_s = "%sx %" max "-s %s\n\n"; for (i in lines) {if (lines[i]["count"] > 2 && lines[i]["count"] < 99) {for (j = 0; j < lines[i]["count"]; j++) {s = s lines[i]["NR"][j] ", ";} s = substr(s, 1, length(s) - 2); printf(fmt_s, lines[i]["count"], i, "\n" s ); s = "";}}}' *.srt | awk -F, '$2 < $1+13' | grep -E -A1 '^[0-9]{,2}x' --color=always | less -r

www.tg-me.com/us/telegram/com.geektips/811

70 viewsedited  Jun 18, 2023 at 04:49

Posting again as I've updated the awk script. Much better now only prints line numbers where the lines are 3 or 4 subtitle lines a way meaning much more likely to be hallucinating or repeating segments.

So found an awk script that does what I just turned into into a one-liner that's all

repeatsubs

for WebVTT subtitles

awk '{ x = lines[$0]["count"]++; lines[$0]["NR"][x] = NR; }END {fmt_s = "%sx %" max "-s %s\n\n"; for (i in lines) {if (lines[i]["count"] > 2 && lines[i]["count"] < 99) {for (j = 0; j < lines[i]["count"]; j++) {s = s lines[i]["NR"][j] ", ";} s = substr(s, 1, length(s) - 2); printf(fmt_s, lines[i]["count"], i, "\n" s ); s = "";}}}' *.vtt | awk -F, '$2 < $1+13' | grep -E -A1 '^[0-9]{,2}x' --color=always | less -r

for SRT subtitles

awk '{ x = lines[$0]["count"]++; lines[$0]["NR"][x] = NR; }END {fmt_s = "%sx %" max "-s %s\n\n"; for (i in lines) {if (lines[i]["count"] > 2 && lines[i]["count"] < 99) {for (j = 0; j < lines[i]["count"]; j++) {s = s lines[i]["NR"][j] ", ";} s = substr(s, 1, length(s) - 2); printf(fmt_s, lines[i]["count"], i, "\n" s ); s = "";}}}' *.srt | awk -F, '$2 < $1+13' | grep -E -A1 '^[0-9]{,2}x' --color=always | less -r

BY GeekTips