Experiment context
Hypothesis. Pinned USP headlines were expected to improve CTR by drawing attention and improve CR by pre-qualifying users before the click. The economic target was lower CPA despite possible CPC growth.
How the test ran. Bidding and targeting were held constant to isolate the creative effect; weaker directions were used first to limit risk.
Traffic split. 50/50.
Performance results
N/A means the metric cannot be calculated because the baseline is zero or the metric does not apply; — means no source value was provided.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 96,729 | 4,510 | 4.66% | 1.97$ | 8,891.39$ | 233.49 | 38.08$ | 5.18% | 87.57% | 90.69% | 103.56% | 106.04% | 96.17% | 87.97% | 109.32% | 97.00% |
| Old ad format | 110,459 | 4,973 | 4.50% | 1.86$ | 9,245.65$ | 265.41 | 34.83$ | 5.34% | — | — | — | — | — | — | — | — |
Result. Unsuccessful / insufficient evidence: traffic fell 13%, CTR was broadly flat, and CPA increased by about 10%; retain the control format.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 6,771 | 987 | 14.58% | 2.02$ | 1,990.74$ | 180.56 | 11.03$ | 18.29% | 95.73% | 89.81% | 93.81% | 101.15% | 90.84% | 113.24% | 80.22% | 126.09% |
| Old Ad Format | 7,073 | 1,099 | 15.54% | 1.99$ | 2,191.51$ | 159.45 | 13.74$ | 14.51% | — | — | — | — | — | — | — | — |
Result. Successful: CR increased 26% and CPA decreased 20% for service3; roll out the pinned format.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 9,897 | 618 | 6.24% | 2.00$ | 1,233.81$ | 43.15 | 28.59$ | 6.98% | 50.73% | 49.48% | 97.54% | 107.50% | 53.19% | 79.31% | 67.07% | 160.28% |
| Old Ad Format | 19,511 | 1,249 | 6.40% | 1.86$ | 2,319.55$ | 54.41 | 42.63$ | 4.36% | — | — | — | — | — | — | — | — |
Result. Successful: CR increased 60% and CPA decreased 33% for service4 despite lower traffic; roll out the pinned format.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 3,329 | 465 | 13.97% | 1.71$ | 796.37$ | 62.97 | 12.65$ | 13.54% | 124.78% | 151.96% | 121.79% | 105.53% | 160.36% | 151.19% | 106.07% | 99.49% |
| Old Ad Format | 2,668 | 306 | 11.47% | 1.62$ | 496.60$ | 41.65 | 11.92$ | 13.61% | — | — | — | — | — | — | — | — |
Result. Unsuccessful for service3: CTR increased 21%, but higher CPC left CR flat and CPA 6% higher; retain the control or retest with lower bids.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 7,542 | 539 | 7.15% | 2.16$ | 1,162.45$ | 21.91 | 53.06$ | 4.06% | 81.15% | 85.15% | 104.93% | 101.63% | 86.54% | 120.25% | 71.97% | 141.22% |
| Old Ad Format | 9,294 | 633 | 6.81% | 2.12$ | 1,343.24$ | 18.22 | 73.72$ | 2.88% | — | — | — | — | — | — | — | — |
Result. Successful for service4: CR increased 60% and CPA decreased 33%; roll out selectively.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 15,496 | 1,926 | 12.43% | 2.12$ | 4,085.69$ | 109.19 | 37.42$ | 5.67% | 237.63% | 269.75% | 113.51% | 96.60% | 260.57% | 339.94% | 76.65% | 126.02% |
| Old Ad Format | 6,521 | 714 | 10.95% | 2.20$ | 1,567.96$ | 32.12 | 48.82$ | 4.50% | — | — | — | — | — | — | — | — |
Result. Successful for service5: conversions increased 240% and CPA decreased 44%; roll out.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 7,004 | 496 | 7.08% | 2.50$ | 1,239.92$ | 35.19 | 35.24$ | 7.09% | 103.18% | 80.39% | 77.91% | 99.38% | 79.89% | 82.53% | 96.80% | 102.66% |
| Old Ad Format | 6,788 | 617 | 9.09% | 2.52$ | 1,552.08$ | 42.64 | 36.40$ | 6.91% | — | — | — | — | — | — | — | — |
Result. Unsuccessful for service4: CTR fell 24%, conversions fell 57%, and CPA increased 45%; retain the control.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 1,549 | 208 | 13.43% | 1.64$ | 341.23$ | 14.89 | 22.92$ | 7.16% | 61.01% | 55.91% | 91.65% | 106.11% | 59.33% | 33.38% | 177.76% | 59.70% |
| Old Ad Format | 2,539 | 372 | 14.65% | 1.55$ | 575.11$ | 44.61 | 12.89$ | 11.99% | — | — | — | — | — | — | — | — |
Result. Unsuccessful for service5: conversions were 2.2× lower and CPA was 2.7× higher; retain the control.
| Label | Impr. | Interactions | Interaction rate | Avg. CPC | Cost | Conversions | Cost / conv. | Conv. rate | Impr new/old | Clicks new/old | CTR new/old | CPC new/old | Cost new/old | Conversions new/old | CPA new/old | CR new/old |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Ad Format | 525 | 52 | 9.90% | 1.44$ | 74.93$ | 0 | N/A | 0.00% | 71.23% | 65.82% | 92.40% | 103.22% | 67.95% | 0.00% | N/A | 0.00% |
| Old Ad Format | 737 | 79 | 10.72% | 1.40$ | 110.28$ | 6.50 | 16.97$ | 8.23% | — | — | — | — | — | — | — | — |
Result. Successful for service4: CR increased 28% and CPA decreased 29%; roll out only for this direction.
Conclusions
Overall verdict: mixed. Pinned headlines should not be rolled out account-wide. They improved conversion efficiency for project-beta service3/service4, project-epsilon service4, project-zeta service5 and project-eta service4, but hurt project-epsilon service3, project-zeta service4 and project-eta service5. Use direction-level rollout decisions and retain the control format where CPA increased.