compare_mt.py Analysis Report

Aggregate Scores

BLEU
	PBMT	NMT	Win?
BLEU	0.2243	0.2403	s2>s1
	[0.2170,0.2315]	[0.2331,0.2479]	p=0.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||ccc}
 & PBMT & NMT & Win? \\ \hline \hline
BLEU & 0.2243 & 0.2403 & s2>s1 \\
 & [0.2170,0.2315] & [0.2331,0.2479] & p=0.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table001}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{001-score-bleu.pdf}
  \caption{Score Comparison}
  \label{fig:001-score-bleu}
\end{figure}

length ratio
	PBMT	NMT	Win?
length ratio	0.9479 (ref=48183, out=45672)	0.9382 (ref=48183, out=45207)	s1>s2
	[0.9409,0.9550]	[0.9296,0.9467]	p=0.0030

\begin{table}[t]
  \centering
  \begin{tabular}{c||ccc}
 & PBMT & NMT & Win? \\ \hline \hline
length ratio & 0.9479 (ref=48183, out=45672) & 0.9382 (ref=48183, out=45207) & s1>s2 \\
 & [0.9409,0.9550] & [0.9296,0.9467] & p=0.0030 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table002}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{002-score-lengthrat.pdf}
  \caption{Score Comparison}
  \label{fig:002-score-lengthrat}
\end{figure}

Word Accuracies

Word fmeas by frequency bucket
frequency	PBMT	NMT
<1	0.1005	0.0493
1	0.2226	0.0850
2	0.3430	0.1735
3	0.3644	0.2408
4	0.4364	0.1667
[5,10)	0.3664	0.2012
[10,100)	0.4843	0.3880
[100,1000)	0.5482	0.5160
>=1000	0.6377	0.6485

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
frequency & PBMT & NMT \\ \hline \hline
<1 & 0.1005 & 0.0493 \\
1 & 0.2226 & 0.0850 \\
2 & 0.3430 & 0.1735 \\
3 & 0.3644 & 0.2408 \\
4 & 0.4364 & 0.1667 \\
[5,10) & 0.3664 & 0.2012 \\
[10,100) & 0.4843 & 0.3880 \\
[100,1000) & 0.5482 & 0.5160 \\
>=1000 & 0.6377 & 0.6485 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table003}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{003-wordacc-frequency-fmeas.pdf}
  \caption{Word Accuracy Analysis}
  \label{fig:003-wordacc-frequency-fmeas}
\end{figure}

Word fmeas by labels bucket
labels	PBMT	NMT
CC	0.7877	0.8043
DT	0.5252	0.5634
IN	0.5360	0.5441
JJ	0.4991	0.4620
NN	0.5485	0.4828
NNP	0.6297	0.4930
NNS	0.5591	0.4992
PRP	0.6274	0.6584
RB	0.4861	0.4662
TO	0.6246	0.6360
VB	0.4720	0.4366
VBP	0.4339	0.4533
VBZ	0.4814	0.5031
other	0.6716	0.6661

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
labels & PBMT & NMT \\ \hline \hline
CC & 0.7877 & 0.8043 \\
DT & 0.5252 & 0.5634 \\
IN & 0.5360 & 0.5441 \\
JJ & 0.4991 & 0.4620 \\
NN & 0.5485 & 0.4828 \\
NNP & 0.6297 & 0.4930 \\
NNS & 0.5591 & 0.4992 \\
PRP & 0.6274 & 0.6584 \\
RB & 0.4861 & 0.4662 \\
TO & 0.6246 & 0.6360 \\
VB & 0.4720 & 0.4366 \\
VBP & 0.4339 & 0.4533 \\
VBZ & 0.4814 & 0.5031 \\
other & 0.6716 & 0.6661 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table004}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{004-wordacc-labels-fmeas.pdf}
  \caption{Word Accuracy Analysis}
  \label{fig:004-wordacc-labels-fmeas}
\end{figure}

Word fmeas by numerical labels bucket
numerical labels	PBMT	NMT
<0.25	0.6103	0.6139
[0.25,0.5)	0.5848	0.5773
[0.5,0.75)	0.5438	0.5189
>=0.75	0.5831	0.5627

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
numerical labels & PBMT & NMT \\ \hline \hline
<0.25 & 0.6103 & 0.6139 \\
[0.25,0.5) & 0.5848 & 0.5773 \\
[0.5,0.75) & 0.5438 & 0.5189 \\
>=0.75 & 0.5831 & 0.5627 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table005}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{005-wordacc-numerical labels-fmeas.pdf}
  \caption{Word Accuracy Analysis}
  \label{fig:005-wordacc-numerical labels-fmeas}
\end{figure}

Source Word Accuracies

Word fmeas by frequency bucket
frequency	PBMT	NMT
<1	0.0000	0.0000
1	0.4460	0.4100
2	0.5981	0.5658
3	0.5887	0.5652
4	0.6518	0.6298
[5,10)	0.6447	0.6380
[10,100)	0.6407	0.6508
[100,1000)	0.6000	0.6058
>=1000	0.0000	0.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
frequency & PBMT & NMT \\ \hline \hline
<1 & 0.0000 & 0.0000 \\
1 & 0.4460 & 0.4100 \\
2 & 0.5981 & 0.5658 \\
3 & 0.5887 & 0.5652 \\
4 & 0.6518 & 0.6298 \\
[5,10) & 0.6447 & 0.6380 \\
[10,100) & 0.6407 & 0.6508 \\
[100,1000) & 0.6000 & 0.6058 \\
>=1000 & 0.0000 & 0.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table006}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{006-wordacc-frequency-fmeas.pdf}
  \caption{Source Word Accuracy Analysis}
  \label{fig:006-wordacc-frequency-fmeas}
\end{figure}

Sentence Buckets

bucket type: length, statistic type: BLEU
length	PBMT	NMT
<10	0.2231	0.2532
[10,20)	0.2098	0.2467
[20,30)	0.2316	0.2379
[30,40)	0.2175	0.2318
[40,50)	0.2219	0.2190
[50,60)	0.2331	0.2554
>=60	0.2726	0.2470

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
length & PBMT & NMT \\ \hline \hline
<10 & 0.2231 & 0.2532 \\
[10,20) & 0.2098 & 0.2467 \\
[20,30) & 0.2316 & 0.2379 \\
[30,40) & 0.2175 & 0.2318 \\
[40,50) & 0.2219 & 0.2190 \\
[50,60) & 0.2331 & 0.2554 \\
>=60 & 0.2726 & 0.2470 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table007}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{007-sent-length-bleu.pdf}
  \caption{Sentence Bucket Analysis}
  \label{fig:007-sent-length-bleu}
\end{figure}

bucket type: reference-output length difference, statistic type: count
lengthdiff	PBMT	NMT
<-20	0	0
[-20,-10)	8	12
[-10,-5)	66	67
-5	41	44
-4	73	58
-3	115	97
-2	173	187
-1	263	256
0	394	423
1	330	344
2	296	275
3	219	194
4	127	153
5	107	93
[6,11)	194	182
[11,21)	39	51
>=21	0	9

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
lengthdiff & PBMT & NMT \\ \hline \hline
<-20 & 0 & 0 \\
[-20,-10) & 8 & 12 \\
[-10,-5) & 66 & 67 \\
-5 & 41 & 44 \\
-4 & 73 & 58 \\
-3 & 115 & 97 \\
-2 & 173 & 187 \\
-1 & 263 & 256 \\
0 & 394 & 423 \\
1 & 330 & 344 \\
2 & 296 & 275 \\
3 & 219 & 194 \\
4 & 127 & 153 \\
5 & 107 & 93 \\
[6,11) & 194 & 182 \\
[11,21) & 39 & 51 \\
>=21 & 0 & 9 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table008}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{008-sent-lengthdiff-count.pdf}
  \caption{Sentence Bucket Analysis}
  \label{fig:008-sent-lengthdiff-count}
\end{figure}

bucket type: sentence-level BLEU, statistic type: count
sentbleu	PBMT	NMT
<0.1	128	116
[0.1,0.2)	832	798
[0.2,0.3)	651	604
[0.3,0.4)	391	396
[0.4,0.5)	191	212
[0.5,0.6)	106	129
[0.6,0.7)	52	63
[0.7,0.8)	32	47
[0.8,0.9)	31	36
>=0.9	31	44

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
sentbleu & PBMT & NMT \\ \hline \hline
<0.1 & 128 & 116 \\
[0.1,0.2) & 832 & 798 \\
[0.2,0.3) & 651 & 604 \\
[0.3,0.4) & 391 & 396 \\
[0.4,0.5) & 191 & 212 \\
[0.5,0.6) & 106 & 129 \\
[0.6,0.7) & 52 & 63 \\
[0.7,0.8) & 32 & 47 \\
[0.8,0.9) & 31 & 36 \\
>=0.9 & 31 & 44 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table009}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics{009-sent-sentbleu-count.pdf}
  \caption{Sentence Bucket Analysis}
  \label{fig:009-sent-sentbleu-count}
\end{figure}

Characteristic N-grams

min_ngram_length=1, max_ngram_length=4

report_length=50, alpha=1.0, compare_type=match

50 n-grams that PBMT had higher match
n-gram	match	PBMT	NMT
phantom	0.9459	34	1
Amy	0.9091	9	0
, who	0.9000	8	0
my mother	0.8889	7	0
And so	0.8571	5	0
else happened	0.8571	5	0
my mother .	0.8571	5	0
something else happened	0.8571	5	0
Avelile	0.8571	5	0
so on	0.8333	4	0
file	0.8333	4	0
rightness	0.8333	4	0
and so on	0.8333	4	0
the fusiform	0.8333	4	0
mother . "	0.8333	4	0
my mother . "	0.8333	4	0
the phantom	0.8333	4	0
Center	0.8333	4	0
fine	0.8333	4	0
fusiform	0.8333	4	0
centers	0.8333	4	0
Oz	0.8333	4	0
mother . " "	0.8333	4	0
viewer	0.8333	4	0
instead	0.8182	8	1
exchange	0.8125	12	2
creativity can	0.8000	3	0
found that	0.8000	3	0
billion pixels	0.8000	3	0
Israel	0.8000	3	0
archive	0.8000	3	0
lose the ability	0.8000	3	0
the Bible	0.8000	3	0
Limpopo	0.8000	3	0
fusiform gyrus	0.8000	3	0
Yeah	0.8000	7	1
pixels	0.8000	3	0
lose the ability to	0.8000	3	0
The point	0.8000	3	0
command	0.8000	3	0
and so on .	0.8000	3	0
excited	0.8000	3	0
like my mother	0.8000	3	0
so on .	0.8000	3	0
exactly like	0.8000	3	0
phantom limbs	0.8000	3	0
Beth	0.8000	3	0
amygdala	0.8000	3	0
address	0.8000	3	0
Galton	0.8000	3	0

\begin{table}[t]
  \centering
  \begin{tabular}{c||ccc}
n-gram & match & PBMT & NMT \\ \hline \hline
phantom & 0.9459 & 34 & 1 \\
Amy & 0.9091 & 9 & 0 \\
, who & 0.9000 & 8 & 0 \\
my mother & 0.8889 & 7 & 0 \\
And so & 0.8571 & 5 & 0 \\
else happened & 0.8571 & 5 & 0 \\
my mother . & 0.8571 & 5 & 0 \\
something else happened & 0.8571 & 5 & 0 \\
Avelile & 0.8571 & 5 & 0 \\
so on & 0.8333 & 4 & 0 \\
file & 0.8333 & 4 & 0 \\
rightness & 0.8333 & 4 & 0 \\
and so on & 0.8333 & 4 & 0 \\
the fusiform & 0.8333 & 4 & 0 \\
mother . " & 0.8333 & 4 & 0 \\
my mother . " & 0.8333 & 4 & 0 \\
the phantom & 0.8333 & 4 & 0 \\
Center & 0.8333 & 4 & 0 \\
fine & 0.8333 & 4 & 0 \\
fusiform & 0.8333 & 4 & 0 \\
centers & 0.8333 & 4 & 0 \\
Oz & 0.8333 & 4 & 0 \\
mother . " " & 0.8333 & 4 & 0 \\
viewer & 0.8333 & 4 & 0 \\
instead & 0.8182 & 8 & 1 \\
exchange & 0.8125 & 12 & 2 \\
creativity can & 0.8000 & 3 & 0 \\
found that & 0.8000 & 3 & 0 \\
billion pixels & 0.8000 & 3 & 0 \\
Israel & 0.8000 & 3 & 0 \\
archive & 0.8000 & 3 & 0 \\
lose the ability & 0.8000 & 3 & 0 \\
the Bible & 0.8000 & 3 & 0 \\
Limpopo & 0.8000 & 3 & 0 \\
fusiform gyrus & 0.8000 & 3 & 0 \\
Yeah & 0.8000 & 7 & 1 \\
pixels & 0.8000 & 3 & 0 \\
lose the ability to & 0.8000 & 3 & 0 \\
The point & 0.8000 & 3 & 0 \\
command & 0.8000 & 3 & 0 \\
and so on . & 0.8000 & 3 & 0 \\
excited & 0.8000 & 3 & 0 \\
like my mother & 0.8000 & 3 & 0 \\
so on . & 0.8000 & 3 & 0 \\
exactly like & 0.8000 & 3 & 0 \\
phantom limbs & 0.8000 & 3 & 0 \\
Beth & 0.8000 & 3 & 0 \\
amygdala & 0.8000 & 3 & 0 \\
address & 0.8000 & 3 & 0 \\
Galton & 0.8000 & 3 & 0 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table010}
\end{table}

50 n-grams that NMT had higher match
n-gram	match	PBMT	NMT
going to show you	0.1250	0	6
going to show	0.1250	0	6
Is	0.1429	0	5
, because the	0.1429	0	5
hemisphere	0.1429	0	5
't even	0.1429	0	5
Is it	0.1429	0	5
'm going to show	0.1429	0	5
: Okay ,	0.1667	0	4
the process	0.1667	0	4
the light	0.1667	0	4
presence of	0.1667	0	4
the camera	0.1667	0	4
the present	0.1667	0	4
process of	0.1667	0	4
: Okay	0.1667	0	4
left hemisphere	0.1667	0	4
years old	0.1667	0	4
the presence of	0.1667	0	4
yet	0.1667	0	4
how do	0.1818	1	8
with a	0.1818	1	8
don 't have to	0.2000	0	3
size of	0.2000	0	3
, all the	0.2000	0	3
it doesn 't	0.2000	0	3
it doesn	0.2000	0	3
, he	0.2000	0	3
AG : Okay ,	0.2000	0	3
I was a	0.2000	0	3
don 't even	0.2000	0	3
if they	0.2000	0	3
not a	0.2000	0	3
core of	0.2000	0	3
core of the	0.2000	0	3
a very early	0.2000	0	3
a little bit	0.2000	0	3
little bit	0.2000	0	3
to show you	0.2000	1	7
penalty	0.2000	0	3
a computer mouse	0.2000	0	3
Everything	0.2000	0	3
, there 's a	0.2000	0	3
death penalty	0.2000	0	3
, as if	0.2000	0	3
the surgeon	0.2000	0	3
AG : Okay	0.2000	0	3
he was	0.2000	0	3
to show	0.2000	1	7
that .	0.2000	0	3

\begin{table}[t]
  \centering
  \begin{tabular}{c||ccc}
n-gram & match & PBMT & NMT \\ \hline \hline
going to show you & 0.1250 & 0 & 6 \\
going to show & 0.1250 & 0 & 6 \\
Is & 0.1429 & 0 & 5 \\
, because the & 0.1429 & 0 & 5 \\
hemisphere & 0.1429 & 0 & 5 \\
't even & 0.1429 & 0 & 5 \\
Is it & 0.1429 & 0 & 5 \\
'm going to show & 0.1429 & 0 & 5 \\
: Okay , & 0.1667 & 0 & 4 \\
the process & 0.1667 & 0 & 4 \\
the light & 0.1667 & 0 & 4 \\
presence of & 0.1667 & 0 & 4 \\
the camera & 0.1667 & 0 & 4 \\
the present & 0.1667 & 0 & 4 \\
process of & 0.1667 & 0 & 4 \\
: Okay & 0.1667 & 0 & 4 \\
left hemisphere & 0.1667 & 0 & 4 \\
years old & 0.1667 & 0 & 4 \\
the presence of & 0.1667 & 0 & 4 \\
yet & 0.1667 & 0 & 4 \\
how do & 0.1818 & 1 & 8 \\
with a & 0.1818 & 1 & 8 \\
don 't have to & 0.2000 & 0 & 3 \\
size of & 0.2000 & 0 & 3 \\
, all the & 0.2000 & 0 & 3 \\
it doesn 't & 0.2000 & 0 & 3 \\
it doesn & 0.2000 & 0 & 3 \\
, he & 0.2000 & 0 & 3 \\
AG : Okay , & 0.2000 & 0 & 3 \\
I was a & 0.2000 & 0 & 3 \\
don 't even & 0.2000 & 0 & 3 \\
if they & 0.2000 & 0 & 3 \\
not a & 0.2000 & 0 & 3 \\
core of & 0.2000 & 0 & 3 \\
core of the & 0.2000 & 0 & 3 \\
a very early & 0.2000 & 0 & 3 \\
a little bit & 0.2000 & 0 & 3 \\
little bit & 0.2000 & 0 & 3 \\
to show you & 0.2000 & 1 & 7 \\
penalty & 0.2000 & 0 & 3 \\
a computer mouse & 0.2000 & 0 & 3 \\
Everything & 0.2000 & 0 & 3 \\
, there 's a & 0.2000 & 0 & 3 \\
death penalty & 0.2000 & 0 & 3 \\
, as if & 0.2000 & 0 & 3 \\
the surgeon & 0.2000 & 0 & 3 \\
AG : Okay & 0.2000 & 0 & 3 \\
he was & 0.2000 & 0 & 3 \\
to show & 0.2000 & 1 & 7 \\
that . & 0.2000 & 0 & 3 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table011}
\end{table}

min_ngram_length=1, max_ngram_length=4

report_length=50, alpha=1.0, compare_type=match

ref_labels=example/ted.ref.eng.tag, out0_labels=example/ted.sys1.eng.tag, out1_labels=example/ted.sys2.eng.tag,

50 n-grams that PBMT had higher match
n-gram	match	PBMT	NMT
, RB RB	0.8571	5	0
WDT VBZ IN	0.8571	5	0
NN RB VBD	0.8571	5	0
WDT NN	0.8571	5	0
VBG NN	0.8571	5	0
RP DT	0.8333	4	0
NN . DT	0.8333	4	0
NN NN ,	0.8333	9	1
RB IN .	0.8333	4	0
NNS VBD JJ	0.8000	3	0
NNP , PRP$ NN	0.8000	3	0
NN IN NNS RB	0.8000	3	0
NNS , RB RB	0.8000	3	0
WP ,	0.8000	3	0
RB IN DT JJ	0.8000	3	0
VBD PRP ,	0.8000	3	0
, WDT VBZ IN	0.8000	3	0
NNS CC VB	0.8000	3	0
JJS NN IN	0.8000	3	0
VBP DT NN TO	0.8000	3	0
JJ MD	0.8000	3	0
NN , NN CC	0.8000	3	0
VBD DT JJ NN	0.8000	3	0
JJR JJ	0.8000	3	0
NN VBD DT NN	0.8000	3	0
IN DT NNS .	0.8000	3	0
, NN , NN	0.8000	3	0
IN NNS RB	0.8000	3	0
PRP$ NN . CC	0.8000	3	0
VB . VB	0.8000	3	0
JJ NN RB	0.8000	3	0
CD CD NNS .	0.8000	3	0
VBG IN NNS	0.8000	3	0
CC RB IN .	0.8000	3	0
RP DT NN	0.8000	3	0
. RB	0.7778	6	1
VBD NNP	0.7778	6	1
NN CC NN .	0.7778	6	1
DT NN IN VBG	0.7500	2	0
CD NNP	0.7500	2	0
CD NN WDT	0.7500	2	0
NNS WDT MD	0.7500	2	0
NN VBZ WP	0.7500	2	0
, WP VBP	0.7500	2	0
NN : DT NN	0.7500	2	0
VBP JJ NNP	0.7500	2	0
PRP VBP NN .	0.7500	2	0
IN IN NNP	0.7500	2	0
IN TO VB	0.7500	2	0
, PRP RB	0.7500	2	0

\begin{table}[t]
  \centering
  \begin{tabular}{c||ccc}
n-gram & match & PBMT & NMT \\ \hline \hline
, RB RB & 0.8571 & 5 & 0 \\
WDT VBZ IN & 0.8571 & 5 & 0 \\
NN RB VBD & 0.8571 & 5 & 0 \\
WDT NN & 0.8571 & 5 & 0 \\
VBG NN & 0.8571 & 5 & 0 \\
RP DT & 0.8333 & 4 & 0 \\
NN . DT & 0.8333 & 4 & 0 \\
NN NN , & 0.8333 & 9 & 1 \\
RB IN . & 0.8333 & 4 & 0 \\
NNS VBD JJ & 0.8000 & 3 & 0 \\
NNP , PRP$ NN & 0.8000 & 3 & 0 \\
NN IN NNS RB & 0.8000 & 3 & 0 \\
NNS , RB RB & 0.8000 & 3 & 0 \\
WP , & 0.8000 & 3 & 0 \\
RB IN DT JJ & 0.8000 & 3 & 0 \\
VBD PRP , & 0.8000 & 3 & 0 \\
, WDT VBZ IN & 0.8000 & 3 & 0 \\
NNS CC VB & 0.8000 & 3 & 0 \\
JJS NN IN & 0.8000 & 3 & 0 \\
VBP DT NN TO & 0.8000 & 3 & 0 \\
JJ MD & 0.8000 & 3 & 0 \\
NN , NN CC & 0.8000 & 3 & 0 \\
VBD DT JJ NN & 0.8000 & 3 & 0 \\
JJR JJ & 0.8000 & 3 & 0 \\
NN VBD DT NN & 0.8000 & 3 & 0 \\
IN DT NNS . & 0.8000 & 3 & 0 \\
, NN , NN & 0.8000 & 3 & 0 \\
IN NNS RB & 0.8000 & 3 & 0 \\
PRP$ NN . CC & 0.8000 & 3 & 0 \\
VB . VB & 0.8000 & 3 & 0 \\
JJ NN RB & 0.8000 & 3 & 0 \\
CD CD NNS . & 0.8000 & 3 & 0 \\
VBG IN NNS & 0.8000 & 3 & 0 \\
CC RB IN . & 0.8000 & 3 & 0 \\
RP DT NN & 0.8000 & 3 & 0 \\
. RB & 0.7778 & 6 & 1 \\
VBD NNP & 0.7778 & 6 & 1 \\
NN CC NN . & 0.7778 & 6 & 1 \\
DT NN IN VBG & 0.7500 & 2 & 0 \\
CD NNP & 0.7500 & 2 & 0 \\
CD NN WDT & 0.7500 & 2 & 0 \\
NNS WDT MD & 0.7500 & 2 & 0 \\
NN VBZ WP & 0.7500 & 2 & 0 \\
, WP VBP & 0.7500 & 2 & 0 \\
NN : DT NN & 0.7500 & 2 & 0 \\
VBP JJ NNP & 0.7500 & 2 & 0 \\
PRP VBP NN . & 0.7500 & 2 & 0 \\
IN IN NNP & 0.7500 & 2 & 0 \\
IN TO VB & 0.7500 & 2 & 0 \\
, PRP RB & 0.7500 & 2 & 0 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table012}
\end{table}

50 n-grams that NMT had higher match
n-gram	match	PBMT	NMT
'' RB	0.1111	0	7
PRP VBP '' RB	0.1250	0	6
VBP '' RB	0.1250	0	6
VBG TO VB PRP	0.1250	0	6
IN NN VBZ	0.1429	0	5
WP VBZ VBG	0.1429	0	5
NN WRB PRP	0.1429	0	5
PRP VBD DT NN	0.1429	0	5
NNS PRP	0.1667	0	4
PRP , CC	0.1667	0	4
TO JJ	0.1667	0	4
IN PRP VBP VBG	0.1667	0	4
WRB JJ NNS	0.1667	0	4
'' RB VBP	0.1667	0	4
WP VBZ VBG IN	0.1667	0	4
VBP '' RB VBP	0.1667	0	4
IN DT NN VBD	0.1667	0	4
NN PRP MD	0.1667	0	4
JJ NNP NN	0.1667	0	4
VBP NNS VBP TO	0.2000	0	3
DT NN , WDT	0.2000	0	3
VBD DT RB	0.2000	0	3
PRP VBD PRP$	0.2000	0	3
RB PRP VBP VBG	0.2000	0	3
NN IN PRP VBZ	0.2000	0	3
VBN PRP	0.2000	0	3
DT NNS , PRP	0.2000	0	3
NNS IN NN .	0.2000	0	3
NNP : NNP ,	0.2000	0	3
JJ , PRP	0.2000	0	3
CD , PRP	0.2000	1	7
IN PRP VBD PRP	0.2000	0	3
NNS CC NNS IN	0.2000	0	3
NNP : NNP	0.2000	0	3
VBD PRP DT	0.2000	0	3
VBD DT RB JJ	0.2000	0	3
VBG TO VB DT	0.2000	0	3
VBD VBG	0.2000	2	11
IN CD NNS IN	0.2000	0	3
IN PRP VBD DT	0.2000	0	3
NNS PRP VBP	0.2000	0	3
PRP MD RB VB	0.2000	0	3
NN VBZ PRP	0.2000	0	3
, PDT DT	0.2000	0	3
JJ , CC PRP	0.2000	0	3
PDT DT JJ	0.2000	0	3
NNS VBP VBN	0.2000	0	3
JJ NNP NN PRP	0.2000	0	3
NNP NN PRP	0.2000	0	3
VB DT NN TO	0.2000	0	3

\begin{table}[t]
  \centering
  \begin{tabular}{c||ccc}
n-gram & match & PBMT & NMT \\ \hline \hline
'' RB & 0.1111 & 0 & 7 \\
PRP VBP '' RB & 0.1250 & 0 & 6 \\
VBP '' RB & 0.1250 & 0 & 6 \\
VBG TO VB PRP & 0.1250 & 0 & 6 \\
IN NN VBZ & 0.1429 & 0 & 5 \\
WP VBZ VBG & 0.1429 & 0 & 5 \\
NN WRB PRP & 0.1429 & 0 & 5 \\
PRP VBD DT NN & 0.1429 & 0 & 5 \\
NNS PRP & 0.1667 & 0 & 4 \\
PRP , CC & 0.1667 & 0 & 4 \\
TO JJ & 0.1667 & 0 & 4 \\
IN PRP VBP VBG & 0.1667 & 0 & 4 \\
WRB JJ NNS & 0.1667 & 0 & 4 \\
'' RB VBP & 0.1667 & 0 & 4 \\
WP VBZ VBG IN & 0.1667 & 0 & 4 \\
VBP '' RB VBP & 0.1667 & 0 & 4 \\
IN DT NN VBD & 0.1667 & 0 & 4 \\
NN PRP MD & 0.1667 & 0 & 4 \\
JJ NNP NN & 0.1667 & 0 & 4 \\
VBP NNS VBP TO & 0.2000 & 0 & 3 \\
DT NN , WDT & 0.2000 & 0 & 3 \\
VBD DT RB & 0.2000 & 0 & 3 \\
PRP VBD PRP$ & 0.2000 & 0 & 3 \\
RB PRP VBP VBG & 0.2000 & 0 & 3 \\
NN IN PRP VBZ & 0.2000 & 0 & 3 \\
VBN PRP & 0.2000 & 0 & 3 \\
DT NNS , PRP & 0.2000 & 0 & 3 \\
NNS IN NN . & 0.2000 & 0 & 3 \\
NNP : NNP , & 0.2000 & 0 & 3 \\
JJ , PRP & 0.2000 & 0 & 3 \\
CD , PRP & 0.2000 & 1 & 7 \\
IN PRP VBD PRP & 0.2000 & 0 & 3 \\
NNS CC NNS IN & 0.2000 & 0 & 3 \\
NNP : NNP & 0.2000 & 0 & 3 \\
VBD PRP DT & 0.2000 & 0 & 3 \\
VBD DT RB JJ & 0.2000 & 0 & 3 \\
VBG TO VB DT & 0.2000 & 0 & 3 \\
VBD VBG & 0.2000 & 2 & 11 \\
IN CD NNS IN & 0.2000 & 0 & 3 \\
IN PRP VBD DT & 0.2000 & 0 & 3 \\
NNS PRP VBP & 0.2000 & 0 & 3 \\
PRP MD RB VB & 0.2000 & 0 & 3 \\
NN VBZ PRP & 0.2000 & 0 & 3 \\
, PDT DT & 0.2000 & 0 & 3 \\
JJ , CC PRP & 0.2000 & 0 & 3 \\
PDT DT JJ & 0.2000 & 0 & 3 \\
NNS VBP VBN & 0.2000 & 0 & 3 \\
JJ NNP NN PRP & 0.2000 & 0 & 3 \\
NNP NN PRP & 0.2000 & 0 & 3 \\
VB DT NN TO & 0.2000 & 0 & 3 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table013}
\end{table}

Sentence Examples

10 sentences where PBMT>NMT at sentence-level BLEU

	Output	sentbleu
Src	To je vzrušujúce samo o sebe.
Ref	That 's exciting in itself .
PBMT	That 's exciting in itself .	1.0000
NMT	This is a thrill of self .	0.1858

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & To je vzrušujúce samo o sebe. &  \\
Ref & That 's exciting in itself . &  \\
PBMT & That 's exciting in itself . & 1.0000 \\
NMT & This is a thrill of self . & 0.1858 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table014}
\end{table}

	Output	sentbleu
Src	A žili spolu šťastne až do smrti.
Ref	And they lived together happily ever after .
PBMT	And they lived together happily ever after .	1.0000
NMT	And he lived with happily to death .	0.2016

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & A žili spolu šťastne až do smrti. &  \\
Ref & And they lived together happily ever after . &  \\
PBMT & And they lived together happily ever after . & 1.0000 \\
NMT & And he lived with happily to death . & 0.2016 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table015}
\end{table}

	Output	sentbleu
Src	Beth Israel je v Bostone.
Ref	Beth Israel 's in Boston .
PBMT	Beth Israel 's in Boston .	1.0000
NMT	Beat Isaill is in Boston .	0.4111

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Beth Israel je v Bostone. &  \\
Ref & Beth Israel 's in Boston . &  \\
PBMT & Beth Israel 's in Boston . & 1.0000 \\
NMT & Beat Isaill is in Boston . & 0.4111 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table016}
\end{table}

	Output	sentbleu
Src	(Potlesk) Posledná otázka, Julian.
Ref	( Applause ) Last question , Julian .
PBMT	( Applause ) Last question , Julian .	1.0000
NMT	( Applause ) The last question , Julian . Julan .	0.4240

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & (Potlesk) Posledná otázka, Julian. &  \\
Ref & ( Applause ) Last question , Julian . &  \\
PBMT & ( Applause ) Last question , Julian . & 1.0000 \\
NMT & ( Applause ) The last question , Julian . Julan . & 0.4240 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table017}
\end{table}

	Output	sentbleu
Src	Zistili sme, že aj bohaté deti z predmestia naozaj chcú DryBath. (smiech) Aspoň raz za týždeň.
Ref	We realized that even rich kids from the suburbs really want DryBath . ( Laughter ) At least once a week .
PBMT	And we found that even rich kids a suburbs really want DryBath . ( Laughter ) At least once a week .	0.7378
NMT	We 've learned that the rich kids have actually wanted DryBothth , ( Laughter ) at the time for a week .	0.1786

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Zistili sme, že aj bohaté deti z predmestia naozaj chcú DryBath. (smiech) Aspoň raz za týždeň. &  \\
Ref & We realized that even rich kids from the suburbs really want DryBath . ( Laughter ) At least once a week . &  \\
PBMT & And we found that even rich kids a suburbs really want DryBath . ( Laughter ) At least once a week . & 0.7378 \\
NMT & We 've learned that the rich kids have actually wanted DryBothth , ( Laughter ) at the time for a week . & 0.1786 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table018}
\end{table}

	Output	sentbleu
Src	JA: Nie som si istý o incidente.
Ref	JA : I 'm not sure about the incident .
PBMT	JA : I 'm not sure about the incident .	1.0000
NMT	J : I 'm not sure of you .	0.4808

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & JA: Nie som si istý o incidente. &  \\
Ref & JA : I 'm not sure about the incident . &  \\
PBMT & JA : I 'm not sure about the incident . & 1.0000 \\
NMT & J : I 'm not sure of you . & 0.4808 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table019}
\end{table}

	Output	sentbleu
Src	Toto bolo opísané nádherne v roku 2006 v knihe, ktorú napísali Michael Porter a Elizabeth Teisberg.
Ref	This was described beautifully in a book in 2006 by Michael Porter and Elizabeth Teisberg .
PBMT	This was described beautifully in 2006 in a book that wrote Michael Porter and Elizabeth Teisberg .	0.6254
NMT	This was originally gorgeous in 2006 in 2006 in 2006 , and they wrote Michael Porrter and Edzabet Tode .	0.1171

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Toto bolo opísané nádherne v roku 2006 v knihe, ktorú napísali Michael Porter a Elizabeth Teisberg. &  \\
Ref & This was described beautifully in a book in 2006 by Michael Porter and Elizabeth Teisberg . &  \\
PBMT & This was described beautifully in 2006 in a book that wrote Michael Porter and Elizabeth Teisberg . & 0.6254 \\
NMT & This was originally gorgeous in 2006 in 2006 in 2006 , and they wrote Michael Porrter and Edzabet Tode . & 0.1171 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table020}
\end{table}

	Output	sentbleu
Src	Prečo je to tak?
Ref	Why is that ?
PBMT	Why is that ?	1.0000
NMT	Why is this ?	0.5081

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Prečo je to tak? &  \\
Ref & Why is that ? &  \\
PBMT & Why is that ? & 1.0000 \\
NMT & Why is this ? & 0.5081 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table021}
\end{table}

	Output	sentbleu
Src	Nemyslím tým doslova 24 hodín, 7 dní v týždni.
Ref	I don 't literally mean 24 hours , seven days a week .
PBMT	I don 't mean literally 24 hours , seven days a week .	0.7272
NMT	I don 't mean literally 24 hours in the week .	0.2525

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Nemyslím tým doslova 24 hodín, 7 dní v týždni. &  \\
Ref & I don 't literally mean 24 hours , seven days a week . &  \\
PBMT & I don 't mean literally 24 hours , seven days a week . & 0.7272 \\
NMT & I don 't mean literally 24 hours in the week . & 0.2525 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table022}
\end{table}

	Output	sentbleu
Src	Rozdiely boli dramatické.
Ref	The differences were dramatic .
PBMT	The differences were dramatic .	1.0000
NMT	The variations were dramatic .	0.5373

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Rozdiely boli dramatické. &  \\
Ref & The differences were dramatic . &  \\
PBMT & The differences were dramatic . & 1.0000 \\
NMT & The variations were dramatic . & 0.5373 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table023}
\end{table}

10 sentences where NMT>PBMT at sentence-level BLEU

	Output	sentbleu
Src	Drobné vychytávky môžu viesť k veľkým zmenám.
Ref	Tiny tweaks can lead to big changes .
PBMT	Now , this idea of marginal raisestas can lead to a big change to happen .	0.1584
NMT	Facferences can lead to big changes .	0.7290

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Drobné vychytávky môžu viesť k veľkým zmenám. &  \\
Ref & Tiny tweaks can lead to big changes . &  \\
PBMT & Now , this idea of marginal raisestas can lead to a big change to happen . & 0.1584 \\
NMT & Facferences can lead to big changes . & 0.7290 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table024}
\end{table}

	Output	sentbleu
Src	Takže tu je video.
Ref	So here 's a video .
PBMT	So here 's the video . "	0.4111
NMT	So here 's a video .	1.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Takže tu je video. &  \\
Ref & So here 's a video . &  \\
PBMT & So here 's the video . " & 0.4111 \\
NMT & So here 's a video . & 1.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table025}
\end{table}

	Output	sentbleu
Src	Nejde o tie čísla.
Ref	It 's not the numbers .
PBMT	It 's not about those numbers .	0.4111
NMT	It 's not the numbers .	1.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Nejde o tie čísla. &  \\
Ref & It 's not the numbers . &  \\
PBMT & It 's not about those numbers . & 0.4111 \\
NMT & It 's not the numbers . & 1.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table026}
\end{table}

	Output	sentbleu
Src	Tu vám teda ukážem dva zábery.
Ref	So here I 'm going to show you two tandem clips .
PBMT	Here you 're going to two footage .	0.1522
NMT	So here I 'm going to show you two footage .	0.7547

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Tu vám teda ukážem dva zábery. &  \\
Ref & So here I 'm going to show you two tandem clips . &  \\
PBMT & Here you 're going to two footage . & 0.1522 \\
NMT & So here I 'm going to show you two footage . & 0.7547 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table027}
\end{table}

	Output	sentbleu
Src	Čo to je?
Ref	What is this ?
PBMT	" What is that ?	0.3861
NMT	What is this ?	1.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Čo to je? &  \\
Ref & What is this ? &  \\
PBMT & " What is that ? & 0.3861 \\
NMT & What is this ? & 1.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table028}
\end{table}

	Output	sentbleu
Src	(potlesk) Je to tak.
Ref	( Applause ) That 's true .
PBMT	( Applause ) It 's so .	0.3826
NMT	( Applause ) That 's true .	1.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & (potlesk) Je to tak. &  \\
Ref & ( Applause ) That 's true . &  \\
PBMT & ( Applause ) It 's so . & 0.3826 \\
NMT & ( Applause ) That 's true . & 1.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table029}
\end{table}

	Output	sentbleu
Src	Svetlo je dobré.
Ref	Light is good .
PBMT	The light 's good .	0.3593
NMT	Light is good .	1.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Svetlo je dobré. &  \\
Ref & Light is good . &  \\
PBMT & The light 's good . & 0.3593 \\
NMT & Light is good . & 1.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table030}
\end{table}

	Output	sentbleu
Src	A to o čom hovorím je toto.
Ref	And what I 'm talking about is this .
PBMT	And that 's what I 'm saying is this .	0.3508
NMT	And what I 'm talking about is this .	1.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & A to o čom hovorím je toto. &  \\
Ref & And what I 'm talking about is this . &  \\
PBMT & And that 's what I 'm saying is this . & 0.3508 \\
NMT & And what I 'm talking about is this . & 1.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table031}
\end{table}

	Output	sentbleu
Src	Zatiaľ sa len učia počítať.
Ref	They 're just learning how to count .
PBMT	Meanwhile just teach to count .	0.3115
NMT	They 're just learning how to count .	1.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Zatiaľ sa len učia počítať. &  \\
Ref & They 're just learning how to count . &  \\
PBMT & Meanwhile just teach to count . & 0.3115 \\
NMT & They 're just learning how to count . & 1.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table032}
\end{table}

	Output	sentbleu
Src	Chlapec: Preto.
Ref	Boy : That 's why .
PBMT	Boy : So .	0.3082
NMT	Boy : That 's why .	1.0000

\begin{table}[t]
  \centering
  \begin{tabular}{c||cc}
 & Output & sentbleu \\ \hline \hline
Src & Chlapec: Preto. &  \\
Ref & Boy : That 's why . &  \\
PBMT & Boy : So . & 0.3082 \\
NMT & Boy : That 's why . & 1.0000 \\
  \end{tabular}
  \caption{Caption}
  \label{tab:table033}
\end{table}