A compilation of the research I've done during my time taking User
Centered Research and Evaluation. Throughout the Spring 2023 semester,
we researched ways to incentivize TikTok users to report biased ads
due to AI algorithms.
As a team of 5, we created a simple yet powerful solution that would
revolutionize the e-bike domain.
Role
UX Researcher
Project
User Centered Research and Evaluation
Timeline
Six Weeks
Tools
Figma, FigJam, Miro
Overview
How does our product incentivize TikTok users to identify biases in
their ads and report them?
Within our group, we decided to focus on ways we can incentivize and
make reporting ads more intuitive for TikTok users. Here, I will focus
more on the contributions I made towards the group project and how my
ideas led to the overall success to the completion of our project. I
helped with the majority of research, interviews, group work, and
design.
Background Research
First, I began with finding some background information on algorithmic
bias. I looked through different definitions I found online and even
cited my own feed. As I perused my social media apps, I realized that
there was a lot more algorithmic bias towards my own preferences. We can
also see algorithmic bias as harmful, which is seen through looking up
the word 'fashion' on Instagram. Looking this up would give us pictures
of predominantely white and skinny women, which perpetuates the idea
that 'fashion' belongs to certain type of person. Although our group
ultimately settled with TikTok's biases within their ads, the initial
background research done gave us good insights into what we should be
particularly focusing on.
Data Analysis
I created the graphs made to help practice analyzing data for future
reports. I also asisted in interpreting these graphs. For example,
from the graphs I made, I derived the hypothesis where I found that
Tweets that had the most likes pertain to Twitter cropping because it
happened on the platform itself. Twitter cropping also occurred
without the consent of the users. However, Portrait AI is done by the
users and the users would often share or retweet other Tweets that are
related to Portrait AI due to popularity and trends.
The Initial Think Aloud Protocol Process
Since this was our first Think Aloud Protocol, we had to first create
the process for it. I assisted in writing the introduction and thought
of the three different tasks the user should go through when doing the
protocol.
Next, we tested out our Think Aloud Protocol. I did this with a user
and with Instagram Reels. At this time, we were still focused on
Instagram and did not make the change to TikTok yet. We decided to
switch to TikTok later on due to TikTok's implicit algorithmic bias
and we became more interested in TikTok's algortihm over Instagram
(and another group in our recitation was doing Instagram haha).
During this Think Aloud, I tested with two Instagram users and they
scrolled through their Instagram reels to see if the ads the received
were biased towards their preferences. Since one user used Instagram
quite a lot, their ads were catered towards things they were
interested in while the other user would gets ads they were not too
interested in.
From the Think Aloud, I found that users were unlikely to report
Instagram Reel ads because there were limited reporting reasons and
that the users would usually just skip the ads instead. I later
observed the same behavior with TikTok ads thus the transition between
the two social media platforms was not too drastic. From this, I found
problems that I aimed to solve.
Walking the Wall
Here, I helped with synthesizing our data together. I came up with
ideas and questions about all of our data when put together. For
example, I stated that "We can make reporting biased ads more
intuitive and work with others (such as those who make algorithms that
decide which ad is shown) to try and make ads more catered and less
harmful towards the user." as a design need.
Project Definition
After synthesizing our data together by walking the wall, we decided to
solidy our project defintion. We asked: "How might we make finding /
reporting biases in TikTok ads that appear because of AI algorithms more
intuitive for everyday users?" I then thought of supporting tasks, such
as interviewing TikTok users and discussing ads that they commonly see
and asking whether the user believes they are biased or not. In addition
to this, users would have to determine whether or not they would
realisticly report the ad during an everyday setting. I then decided
that this project would impact everyday TikTok users, developers of
TikTok, and the people who pay for ads
Research Goals
Although I stepped back for determining the estimated times in which our
research goals will take, I helped with deciding what our research goals
will be. I helped with figuring out the hurdle of inaction. I asked "How
can we incentivize TikTok users to identify biases in their ads and
report them?" From this, I helped with synthesizing smaller questions
and figuring out hypothetical solutions to them.
Interview Guide
This is where we created the baseline for our interiview. In particular,
I helped with creating our interview questions and asking the core
questions. i also decided the general format for the interview plan. We
decided to do another Think Aloud Protocol but with TikTok. We managed
to pilot test the guide with other UCRE students to ensure that our
interview guide is not asking too much and it suffcies for an average
interview about TikTok ad biases.
Conducting the Research Session
This is where I put the interview guide to the test. The person I
reached out to was a student at NJIT who used TikTok occassionally.
Although we were not within eachother's immediate social circle, the
interactions went well as I gained more insight in how the user
interacted with TikTok ads. I learned that some people will immediately
skip over an ad if they noticed that it is tagged with 'sponsored.' I
then took interpretation notes of what I noticed from the reseach
session.
Affinity Diagramming
In my opinion, affinity diagramming took the most time and effort during
our research project. We had to consistently narrow our ideas down and
break them into more concise ideas. However, it was the most rewarding.
It helped me understand our users needs and it helped power my ideas
behind the implementations we were brainstorming to add to TikTok's
interface. I began thinking in the shoes of an average, everyday TikTok
user rather than a researcher.
The Research Report
So this is the semi? big research report! We basically synthesized and
analyzed all of the data we collected as of now (in the timeline of the
website you are following). Within this report, I helped write the
executive summary, such as talking about high level insights. I also
wrote apart of our insights and created a model of a user flow diagram.
The Survey
We practiced designing and conducting a survey to inform our research
project on algorithmic bias auditing. By reviewing and analyzing our
previously tested Think Aloud Users, I helped narrow down our target
demographic. From this, I came up with questions that probed the survey
taker about their thoughts on TikTok ads, the ad biases, how they report
ads, and if those ads were difficult to report. Remember, our aim is to
figure out how to incentivze the users to report more ads that may be
triggering or harmfully biased towards them or others.
We pilot tested the survey with others before fully sending out the
survey. I was able to get a sizeable amount of people to answer our
survey through my connections on Discord and advertising in person to
others I knew in passing.
The Summary Report
We analyzed the data from our survey and came up with results. I helped
with analyzing the data and understanding what the users needed. In
particular, I realized that TikTok does little to handle complaints or
encourage users to report ads. Users would be willing to report ads but
TikTok gives them a non-intuitve way to report ads. This helped give me
and my group ideas on how to solve this issue.
Speed Dating Storyboarding
I made eight individual storyboards and ultimately chose three of them
to show our user testers. Overall, we had fifteen storyboards (three
from each member). The first storyboard I made was safe and confortable
for the user, the second is progressively riskier, and the final one was
intentionally risky, and tested the social boundaries for a social media
app. I then tested all of the storyboards with users to see how they
would feel about these storyboards. From these storyboards, we chose the
ones our users positively responded to the best.
We started creating our prototype. We first decided on what our
prototype will include. We decided to implement one of my ideas of
including a report button that the user can click and it will lead them
directly towards the report page. Another implementation that helped to
add in our prototype was showing visual statistics of what other users
reported the ad for. I then wrote about our riskiest assumptions when
making this prototype as it would mean that users would be incentivized
to report the ad. However, users may not feel the need to and completely
skip over our implementation.
The Lo-Fi Prototype
I decided that we should make our prototype on Figma. By using Figma, it
gives the user the experience of testing the real product. By making it
Lo-Fi, the user understands that this is not a real working prototype,
but only just a prototype. I described our prototype as well as outlined
the ususal task a user would go through when reporting an ad and how our
implementation would help their experience become more intuitive and
efficient.
From our Lo-Fi prototype, I learned that our users found the
'exclamation point' button useful as reporting an ad is only one click
away. One point I noticed was that users would skim the statistics of
others who reported the ad before. From this, I decided that we should
put a heavier emphasis on the report options themselves. We can give the
user some space to write about why that ad is being reported for the
selected reason. We can then hide the statistics and allow it to be a
toggled option if the user wants to know what others reported the ad
for. I helped decided our final changes and the next steps we were
planning on taking.
The Final Research Poster
Reflection
Since i did a sizeable amount of the work within the group (probably a
good 35%), I must have missed something within this blog post. I did
link all the projects to the best of my knowledge so hopefully its
exhaustive!
For the most part, I enjoyed working with my team with this project. It
was an interesting problem statement that was presented to us and we
handled it with care. After taking this course and going through all of
the modules, I believe I learned a lot and everything I've learned in
this course will be applied to my future in research. I'm definitely
excited to start other research projects and maybe, one day, I will come
back to this page and see how far I've progressed (or utilize this
research for another project lol).
