singh_2019_training

phishing-detection psych-201

Participants classified emails as phishing vs. ham across three phases: pre_training (10 emails, 20% phishing, no feedback), training (40 emails with outcome feedback), and post_training (10 emails, 20% phishing, no feedback). Three experiments (exp0: outcome feedback, N=296; exp1: incentive manipulation, N=299; exp2: detail feedback, N=223) manipulated the frequency of phishing emails during training (25/50/75%) between subjects, each drawing from a shared 239-email database. Per email trial participants gave three responses: a binary phishing decision, a confidence rating (50–100), and an intended action (1–6). The research question was whether the frequency of experienced phishing emails during training affects detection performance (hit rates, false-alarm rates, signal-detection d' and criterion) at post-test.

Responses
99,338
9,816 held out
CORE effects
–
no effects simulated
CORE test NLL (avg)
0.654
rank 1 of 7 models
Best model
CORE
test NLL (avg) 0.654

Likelihood of human responses

modelPNLLtest NLL (avg)test responses train NLL (avg)train responses
CORE best6,424.60.65459,8160.619189,522
CORE (no prediction errors)6,604.30.67289,8160.639489,522
CORE (no phonemes)8,472.30.86319,8160.245589,522
CORE (no forgetting)6,917.80.70479,8160.677789,522
CORE (no hierarchy)9,124.40.92959,8160.930089,522
Transformer9,380.20.95569,8160.718289,522
Centaur7,032.50.71649,816––

Experiments

exptask classparticipantsresponsesconditionssimulatedeffects
exp0PhishingTraining29636,6983no–
exp1PhishingTraining29935,8803no–
exp2PhishingTraining22326,7603no–

Behavioral effects

No behavioral effects were simulated for this study.

Transcript

You are taking part in a study on detecting phishing emails. Phishing emails are fraudulent messages that try to trick you into revealing personal information or clicking malicious links. You will be shown a series of emails, one at a time, and you will answer three questions about each:
1. Q1 - Is this a phishing email? Press Y if you think it is phishing, or N if you think it is a legitimate email.
2. Q2 - How confident are you in that answer? Type a number from 50 (not confident at all) to 100 (fully confident).
3. Q3 - What would you do if you received this email? Choose one action: D=Respond to the email; C=Click the link or open the attachment; A=Check the sender; E=Check the link; F=Delete the email; B=Report the email.
During the training phase you will receive feedback on your answers. During the pre-training and post-training phases you will not receive any feedback.
Phase: pre-training.
Email: From: message@adobe.com. Subject: Explore your account. Welcome. We are glad you have joined us.
Your Adobe ID links all of your Adobe experiences under one account, using this email address. Here are some great ways to leverage it.
Pass projects seamlessly between desktop and mobile devices. If you don't have the 2017 release of Creative Cloud, test drive free trials. Get started with Creative Cloud Libraries. Thanks to the power of Adobe CreativeSync, this in-app feature gives you easy access to all of your design assets as you work. Jumpstart your projects with integrated Adobe Typekit fonts.
Check out Adobe Stock's latest collections, advanced search and deep integration with Creative Cloud apps. Try Adobe Stock risk-free today. Create stunning graphics, web pages, and video stories in minutes with Adobe Spark, part of your Creative Cloud subscription. 
Explore my account (https://www.adobe.com)
Thanks,
 Adobe
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Email: From: no-reply@yahoo.com. Subject: Password change for your Yahoo account. Hi Ethan,
The password for your Yahoo account, a**********er, was recently changed.
If you made this change, you're all set.
Didn't change your password?
Please take these steps to secure your account.
Recover your account here:https://login.yahoo.com/forgot? (https://login.yahoo.com/forgot?)
Review your phone numbers and email addresses and remove the ones that don't belong to you.https://login.yahoo.com/account/security? (https://login.yahoo.com/account/security?)
Thanks,
Yahoo
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Email: From: service@remitly.com. Subject: Your Remitly Account has been deleted. Greetings from Remitly.com,
As you requested, we have deactivated your Remitly account. We appreciate your past business and we look forward to you coming back soon.
 If you want to activate your account again please Contact Customer Service (http://www.remitly.com/contact) to reactivate your account . 
Thank you for using our services.
- The Remitly Team
Q1 Is this email phishing? You press N.
Q2 How confident? You press 98.
Q3 What action? You press F.
Email: From: jim.jenkins@global-infra.com. Subject: Revised Desktop Support Service. The executive desktop support services have been streamlined due to recent organizational changes.
Click here (http://www.google.com) to view the revised service offerings.
We appreciate your cooperation in this matter.
Global Infrastructure Team
Q1 Is this email phishing? You press N.
Q2 How confident? You press 78.
Q3 What action? You press F.
Email: From: accounts@online.chase.com. Subject: Your 401k Account. Dear Employee:
Due to the migration of our pension system, we will need you to update your information at Chase Online in order for us to continue making our portion of your payments into your 401k account. This will need to be done before your next scheduled pay period. Failure to do so could change your retirement.
https://chaseonline.chase.com/Logon.aspx (http://Chase-Security.html) 
Sincerely, Pension and Payroll Department.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press C.
Email: From: davidcampbell@gmail.com. Subject: David Campbell's Resume. Hello,
I want to thank you for taking time out of your busy schedule to help me with my job venture. I really admire your company. I would like to learn and become a part of your team.
I look forward to assisting you and where ever you need help.
Click here (http://www.googledocs/david) to view my resume
Thank you for all your help,
David Campbell
Q1 Is this email phishing? You press N.
Q2 How confident? You press 68.
Q3 What action? You press E.
Email: From: help@dropbox.com. Subject: You've successfully reset your Dropbox password. Hi,
Your Dropbox account password was recently reset.
While we've updated your password, any computers or phones that you previously linked to your Dropbox account are still connected. You can disconnect a lost or stolen device from your account settings. (https://www.dropbox.com)
If you changed your password for security reasons, we strongly recommend that you unlink any devices, web sessions, or apps that look unfamiliar or that you're concerned about. See this Help Center article (https://www.dropbox.com) for more information.
If you didn't make this change, please let us know (https://www.dropbox.com/contact).
Thanks!
- The Dropbox Team
Q1 Is this email phishing? You press N.
Q2 How confident? You press 91.
Q3 What action? You press F.
Email: From: mcinfo@ups.com. Subject: Your New UPS My Choice Membership. Welcome to UPS My Choice, Sophia.
UPS My Choice driver Your new membership is ready to use. Manage delivery alerts, request changes, and set preferences to make deliveries more convenient for your schedule.
UPS My Choice Membership Information 
Delivery Address: 
280 North Craig
Apt G4
pittsburgh, PA 15213, US
You can change your preferences or renewal options anytime: Just log into your ups.com (http://www.ups.com) profile. If you'd like a reminder about what your membership includes, check out the full list of benefits.
Thanks for being a great customer. 
Sincerely,
Your UPS My Choice Team
Q1 Is this email phishing? You press N.
Q2 How confident? You press 86.
Q3 What action? You press F.
Email: From: support@micrsoft.com. Subject: Evaluations . The following evaluations have been assigned to you.
Login to portal complete these evaluations.
LINK TO LOGIN USING SECURE ENCRYPTION (http://www.fnhfdcb.com)
BE AWARE: Your login will time out after 60 minutes. Your answers will be lost if you do not click on the "FINISH" button before 60 minutes lapses. There is no prompt when your 60 minute session has expired. Please save extensive comments periodically and check your time.
Elizabeth Hollinger
ITS Account Evaluations
Connected to Microsoft Exchange
© 2014 Microsoft Corporation. All rights reserved.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 96.
Q3 What action? You press C.
Email: From: adorama.com@e.adorama.com. Subject: Your Order was Delivered! It's a great time!. Your order has been delivered
Order: #22067001
UPS has indicated that your package has been delivered! 
At Adorama , YOU are our top priority. In the unfortunate chance that something falls short of your expectations, please contact us via email at custserv@adorama.com or phone 800-223-2500. 
We welcome you to visit us online at Adorama for all your Photography, Pro Video & Pro Audio needs. Thanks for your patronage. 
Carrier: UPS
Tracking #: 1Z0X118A0341368229 (#1Z0X118A0341368229)
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press C.
Phase: training.
Email: From: return@irs.tax.com. Subject: Important information regarding your Tax Refund. *********PLEASE DO NOT RESPOND TO THIS EMAIL *********
We are writing to you because your federal Tax return is available for refund.
For status of your refund and more information, please click here (http://www.google.com)
Sincerely,
Internal Revenue Service
Refund Department
PLEASE NOTE: Do not respond to unsolicited e-mails that claim to come from the IRS. The IRS does not use email to request personal information. Forward suspicious emails to phishingemail@irs.gov (mailto:phishingemail@irs.gov)
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 97.
Q3 What action? You press B.
Feedback: your answer was correct.
Email: From: help@irs.us.gov. Subject: Important information concerning your tax refund for 2017. *********PLEASE DO NOT RESPOND TO THIS EMAIL *********
We are writing to inform you that your Federal Tax payment (ID: 88380290), recently sent is now available for refund.
For your security, new changes to the account listed above may be declined. If applicable, you should advise any Additional Card Member(s) on your account that their new changes may also be declined.
For more information, please click here (http://www.google.com) to see your refund from the IRS.
Your prompt response regarding this matter is appreciated.
Sincerely,
IRS
PLEASE NOTE: Do not respond to unsolicited e-mails that claim to come from the IRS. The IRS does not use email to request personal information. Forward suspicious emails to phishingemail@irs.gov (mailto:phishingemail@irs.gov)
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 96.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: it-help@cmu.edu. Subject: Computing Services News. **To verify the authenticity of this message, visit the Computing Services website at https://www.cmu.edu/computing/ (https://www.cmu.edu/computing)**
Welcome back for the Spring 2019 semester. Take a look at what's new and improved from Computing Services.
Zoom Web Conferencing
Zoom, the preferred CMU web conferencing tool, enables easy-to-use video collaboration with colleagues and those from other organizations.
New Antivirus Software for University-Owned Macs
ESET Endpoint Security antivirus is now available for all university-owned Mac computers. With release of this supported solution, you can now safely upgrade university-owned computers to Mojave!
Exchange and G Suite Calendar Coordination
View both G Suite and Exchange calendar availability for easy meeting coordination.
Classroom Improvements
Classroom Improvements continue as part of the Classroom and Learning Spaces Project.
Online Safety Reminders
 Check your security level- Be aware of scam emails
Visit the Computing Services website often to stay up to date on our latest news and feature stories; OR subscribe to our feature news and service alerts.
Computing Services
Carnegie Mellon University
https://www.cmu.edu/computing/ (https://www.cmu.edu/computing)
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: info@jobs.com. Subject: Hi Jacob, I'm reaching out to you about an opportunity. Hi Jacob, 
 A new and exciting opportunity was just posted near you. Looking at your profile and past searches, we believe the position at Leidos may be a great fit! Learn more about this opportunity below.
Click to View (http://www.google.com/jobs)
Not what you're looking for? 
Check out these other jobs
Computer Security Analyst
Posted Today Near 16217
Click to View (http://www.pointparkuniversity.com)
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 100.
Q3 What action? You press B.
Feedback: your answer was incorrect.
Email: From: leasing.meyers@gmail.com. Subject: Pest Control. Dear Tenants of Webster, 
I hope this finds you well, On Tuesday 12/18/2018 from 11AM-4PM an exterminator will be coming to Webster Towers and will be entering every apartment. We apologize for any inconvenience and we look forward to your full cooperation. 
We have also started online payment for the Webster Towers tenants. Use following link to access the payment site: Mayers Management (http://www.mayersmanagement.com)
Thanks, Meyers Management
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: extracare@pharmacy.cvs.com. Subject: $2 Off + Look At These Great Deals. CVS Pharmacy
ExtraCare
Mike, we picked these deals for you.
Save $2
on your in-store or online Grocery purchase.
Choose snacks, drinks, candy and more.
Get coupon (http://www.pharmacy.cvs.com)
 Print or Send to Card through Monday, 2/18/19.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: blockchain@talentsearch.com. Subject: Blockchain Executive Program January 2019. Hello David, 
Congratulations! 
Based on your application, you have been selected for the Blockchain Executive Program January 2019 Cohort at Appolo University. 
Next Step: ENROLL FOR THE PROGRAM 
You can enroll for the Program using either of the two options below:
Option 1: You can join the program by clicking the link below. The final Program Fee that you need to remit after considering your scholarship eligibility is updated at the link below. Use Option 1 in the payment options to be able to join the Program.
https://www.talentsearch.com/requested-amount.dpl?payment_id=11243&i=2750&t=d334f5aa4b8aee2b46fe6d (https://www.talentsearch.com/requested-amount.dpl?payment_id=11243&i=2750&t=d334f5aa4b8aee2b46fe6d)
You can pay using your own sources or opt for a 9 to 24 EMI Option. For more details, contact your Relationship Manager Sarah @ +1-64589723458
Enrollments are on first-come-first-served basis. As there are limited seats, you are requested to enroll on time. Kindly note that we will not be able to hold the seat as there are other candidates awaiting their turn. 
For any queries, get in touch with your Relationship Manager Sarah @ +1-64589723458
Once again, we are pleased to welcome you to this Program. 
Sincerely,
Appolo University | TalentSearch
Q1 Is this email phishing? You press N.
Q2 How confident? You press 91.
Q3 What action? You press E.
Feedback: your answer was correct.
Email: From: accounts@my.amazon.com. Subject: Your Amazon account status. We have a new product that we offer for free only for a honest review!
Order Details:
Order # 002-527657-2343090
Shipping: FREE Shipping
Order total: $212.34
Please let us know if you are interested!
Q1 Is this email phishing? You press N.
Q2 How confident? You press 95.
Q3 What action? You press F.
Feedback: your answer was incorrect.
Email: From: account@security.paypal.com. Subject: Account Terminated. Your Account will remain locked for your protection! You must resolve this now by visiting the website below and following the instructions. 
Case id : 9000321-128. Login attempt from unknown device. 
Dear Client 
Someone has accessed your account, so we have temporarily locked it to keep your personal informations in safe. To unlock your account, you may need to pass a security check. Note that attempting to access someone else is a violation of PayPals terms. It may also be illegal. To reset your account: 
1-Click on the link PayPal-Security.html (PayPal-Security.html). 
2-Open the page in a browser window secure. 
3-Follow the instructions.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 96.
Q3 What action? You press E.
Feedback: your answer was incorrect.
Email: From: help-check@human.resource.com. Subject: An important email to respond. Hello!
This email is used for attention check during the experiment. It will be used to evaluate your attention in the experiment.If you are reading this email, please do the following actions: 
 1. Select this as a phishing email in the first question (Q1). 
 2. Select the "report this email" option in the third question(Q3). 
If you are not paying attention and select the wrong options for this email, you may not be able to continue the experiment. Your attention is important for the experiment. 
Thanks, and good luck!
Human Resource Management
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 100.
Q3 What action? You press B.
Feedback: your answer was incorrect.
Email: From: account-noreply@adobe.com. Subject: Verification code. Your verification code is: 373949
If you didn't request this please contact us immediately.Contact Us (https://www.adobe.com)
Thanks,
Adobe
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: safety@privacy.chase.com. Subject: Update for you Account. Dear Valued Chase Memeber: 
We are committed to providing the tools you need to ensure the safety of your personal information.
Your account has been recently compromised from a different ip address. For your safety we have blocked access to all of your online accounts. Please access our site through the link below to update your security info.
Click here: https://chaseonline.chase.com/Logon.aspx (http://Chase-Security.html) 
Thank you for being our customer. Your safety is our number one concern.
Sincerely, 
JP Morgan Chase Bank, N.A.
Chase Privacy Operations, P.O. Box 659752, San Antonio, TX 78265-9752. 
© 2016 JPMorgan Chase & Co.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 95.
Q3 What action? You press F.
Feedback: your answer was incorrect.
Email: From: unitedairlines@united.com. Subject: Welcome to MileagePlus. MileagePlus enrollment confirmation for John Smith You are now a member of the award-winning United Airlines MileagePlus program.
Mr. Smith,
RL3894503
Remember to provide your MileagePlus number when you make a flight, car or hotel reservation or use the services of a MileagePlus partner.
For membership help or information, visit united.com/mileageplus (https://www.united.com)
E-mail Information, Please do not reply to this message using the "reply" address.The information contained in this e-mail is intended for the original recipient only.
Notification: MileagePlus Enrollment Confirmation
E-mail Address: smithjohn@gmail.com
MileagePlus Number: RL3894503
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: jeremyp@gmail.com. Subject: Summer Internship Application. To Whom It May Concern:
I am replying to your post about the summer internship.
I am currently pursuing my Masters’ in Accounting at the Kelley School of Business at Indiana University.
I am also pursuing my CPA following my completion of my degree.
I have expertise in Microsoft Access, Excel, Word and Powerpoint, Telnet and Ethernet Configuration.
Here is the link (http://www.jhgvd56sh.com) to my CV and resume.
Thanks for taking the time to look at my application.
Sincerely look forward to joining your team!
Best Regards,
Jeremy Paulson.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 98.
Q3 What action? You press E.
Feedback: your answer was incorrect.
Email: From: password@twitter.com. Subject: Security alert: new or unusual Twitter login. Looks like there was a login attempt from a new device or location 
If this wasn't you: Secure your account by resetting your password now. 
Reset password (https://www.twitter.com)
If this was you: Just to be safe, we may ask you some security questions the next time you sign into your account @mikewilliam. 
How do I know an email is from Twitter? 
Links in this email will start with "https://" and contain "twitter.com." Your browser will also display a padlock icon to let you know a site is secure.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 97.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: no-reply@accounts.google.com. Subject: Help us protect you: Security advice from Google. Add a backup phone as an alternative 2-step Verification step
andrew19@gmail.com
A backup phone number ensures that you can sign in when your main second verification step is unavailable.
Add a backup number and see other personalised recommendations in the Security Check-up.
TAKE ACTION (https://www.gmail.com)
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: jpaulson@gmail.com. Subject: Summer Internship Application. Dear Sir or Madam:
 I am interested in the Summer Internship you posted. I am currently pursuing my Masters’ and want to get CPA upon completion. Expertise in Microsaft suite, Telnet and Ethernet Configuration.
More detailed information on my qualifications can be found here at my CV (http://www.dfke.com)
Thanks for your pay attention.
Sincerely look forward to joining your team!
Best Regards,
Jeremy Paulson.
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 100.
Q3 What action? You press B.
Feedback: your answer was correct.
Email: From: help-check@human.resource.com. Subject: An important email to respond. Hello!
This email is used for attention check during the experiment. It will be used to evaluate your attention in the experiment.If you are reading this email, please do the following actions: 
 1. Select this as a phishing email in the first question (Q1). 
 2. Select the "Delete Email" option in the third question(Q3). 
If you are not paying attention and select the wrong options for this email, you may not be able to continue the experiment. Your attention is important for the experiment. 
Thanks, and good luck!
Human Resource Management
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: delivery@thomascook.com. Subject: Transaction Alerts - ThomasCook Prepaid Card. Dear Customer,
Greetings from Thomas Cook! 
Your transaction of USD 381.76 was declined due to Exceeding Funds Available 
To access your travel card account online, please visit us at www.thomascooktravelcards.com (https://www.thomascooktravelcards.com)
Looking forward for more opportunities to be of service to you.
Sincerely, 
Payment Solutions 
Thomas Cook (USA) Limited
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: noreply@pharmacy.cvs.com. Subject: $5 Coupon + Just for You!. CVS Pharmacy
 ExtraCare
Looking for these, Mia?
Save $5, on your $25 Digestive Health purchase in store or online.
 Get coupon: CVS Pharmacy (https://www.cvspharmacy.com)
Print or Send to Card through Wednesday, 2/13/19. Excludes sale items. Other exclusions apply.
 View weekly ad
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: manager@gmail.com. Subject: work meeting. to all employees, will have a meeting first thing in the morning. please review the notes (http://www.cnbfdsmf.com) for it and be ready to present your portion.
thanks
the boss man
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press F.
Feedback: your answer was incorrect.
Email: From: pkginfo@ups.com. Subject: UPS Status Notification, Tracking Number 1Z0X118A4321976826. Please see below for package information and current transit status.
As a residential package recipient, you may enroll in UPS My Choice to get proactive package alerts on all future packages addressed to you. Where available, you may also get a 4-hour delivery window.
Shipment Details
Tracking Number: 1Z0X118A4221976826 
Status: Delay 
Shipped To: PITTSBURGH, PA, US 
UPS Service: UPS Ground 
Weight: 12.5lbs 
Package Progress
Location Date Local Time Description
Secaucus, NJ, United States 01/02/2019 11:59 PM We've incorrectly sorted this package which may cause a delay.
Secaucus, NJ, United States 01/02/2019 11:58 PM Origin Scan
Tracking results provided by UPS 01/03/2019 9:13 A.M. Eastern Time 
 Track Live Status (http://www.ups.com/livestatus)
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press E.
Feedback: your answer was correct.
Email: From: account@it.amazon.com. Subject: Your Amazon account has been temporarily suspended. Account suspended
108-4596473-8009841 FF 
Hello, 
We were unable to validate important details about your Amazon Web Services (AWS) Account. Your AWS account has been suspended. 
Please follow the link below to reactivate your account.
Repair Link (www.repair.link.com)
Account Details 
Account #108-4596473-8009841 
Amazon IT Team
Conditions of Use Privacy Notice 1996-2015, Amazon.com, Inc. or its affiliates 05/10/2017
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 100.
Q3 What action? You press B.
Feedback: your answer was correct.
Email: From: reactivate@my.amazon.com. Subject: Amazon Account Suspended - Action Required. Hello, 
We were unable to validate important details about your Amazon Web Services (AWS) Account number #108-4596473-8009841. As a result, we have suspended your AWS account for security purposes.
To re-activate your account, please visit your account details to confirm the payment information for your account. 
Update Your Payment Method (http://Amazon-Security.html)
Thanks!
Amazon IT Team
Conditions of Use Privacy Notice 1996-2015, Amazon.com, Inc. or its affiliates 05/10/2017
Q1 Is this email phishing? You press N.
Q2 How confident? You press 98.
Q3 What action? You press C.
Feedback: your answer was incorrect.
Email: From: bowne@customersat.com. Subject: Creative Center Customer Satisfaction Survey. Dear Valued Customer,
The Creative Center is initiating a comprehensive survey of its customers' satisfaction, and would very much like to include your feedback. Bowne Business Solutions is proud to be the onsite provider of your graphic services. I would like to invite you to take a moment to complete a brief survey about the BBS services. Your individual identity is guaranteed to be 100% protected, as all responses are accepted anonymously on a secure Web server, where the survey is hosted. We sincerely hope you will take a few minutes to participate, as your feedback and comments will help BBS better understand your needs.
Click here (http://www.google.com/form/) to go the survey.
If you need assistance, please e-mail bowne@customersat.com.
Your opinions and satisfaction are very important to us. Thank you very much for your feedback!
Sincerely,
John Zilker
Client Services Manager
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 98.
Q3 What action? You press B.
Feedback: your answer was incorrect.
Email: From: message@adobe.com. Subject: Explore your account. Welcome. We are glad you have joined us.
Your Adobe ID links all of your Adobe experiences under one account, using this email address. Here are some great ways to leverage it.
Pass projects seamlessly between desktop and mobile devices. If you don't have the 2017 release of Creative Cloud, test drive free trials. Get started with Creative Cloud Libraries. Thanks to the power of Adobe CreativeSync, this in-app feature gives you easy access to all of your design assets as you work. Jumpstart your projects with integrated Adobe Typekit fonts.
Check out Adobe Stock's latest collections, advanced search and deep integration with Creative Cloud apps. Try Adobe Stock risk-free today. Create stunning graphics, web pages, and video stories in minutes with Adobe Spark, part of your Creative Cloud subscription. 
Explore my account (https://www.adobe.com)
Thanks,
 Adobe
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: communications@em.aetna.com. Subject: Protect your health records on your Aetna member website. Protecting your personal information online
Dear Emily, 
Thank you for signing up for your Aetna member website. (https://www.aetna.com/individuals-families/member-rights-resources/member-services.html) Now you can access your benefits whenever you want.
Simple ways to stay secure
Here are a few tips to help protect your information online.
1. Keep your user name and password in a safe place. This helps ensure nobody else can use your account.
2. Do you use a public location -- such as a library -- to look at your account? Never save your personal information to a public computer.
3. When you're done, make sure you log out of your account and close your browser.
We want to help protect your personal information. If you didn't register for this online account, contact us (https://www.aetna.com/individuals-families/contact-aetna.html) right away.
We're here to help
Contact us by email through your member website at aetna.com.
Call us at the number on the back of your Aetna ID card.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: rewards@online.walmart.com. Subject: You've Won FREE Walmart Coupons!. 17/10/2017
Account No: 108-455294-800125-MN
To our valued Walmart customer, 
The team at Walmart Online Services is happy to announce that you have been chosen as the 'User of the month' lottery winner! This is a monthly event where we randomly choose a customer to receive free Walmart coupons. In order to access your coupons, please visit the link below:
Walmart Reward Coupons (http://walmart/rewards/coupons)
Using any of these coupons will result in a donation to St. Jude's Children Hospital, so you can now receive free items and donate to a good cause at the same time. We hope you'll help support us!
From,
Your Friends at Walmart Online Services
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press F.
Feedback: your answer was incorrect.
Email: From: admissions@paris.edu. Subject: Applying to Paris College of Art. Dear Tushar, 
We know how complicated (and daunting!) applying to university can be, so we thought we'd give you our application process in 5 easy steps: you're welcome!
1. Submit your application form and pay the application fee online at: apply.paris.edu/apply (http://www.apply.paris.edu/apply) As soon as we receive your application you will be assigned an Admissions Counselor, who will be your main contact should you have any questions.
2. After you submit your application you will have access to your Application Status Portal. There you will be able to review your application requirements and submit missing items.
3. Have your official transcripts sent by your school to admissions.paris.edu (http://www.admission.paris.edu) or by post to our mailing address below.
4. Once we have all of your application materials, we will invite you to schedule an your admissions interview. This interview can be conducted online or in-person if you will be in France.
5. Et voila, as we say here in France! All that's left to do is wait and keep an eye on your application status by logging in to the portal. Depending on the deadline you applied by, you will be given a response within a few weeks
Here at PCA we pride ourselves on our fast turnaround, so the sooner you get all your application materials to us, the sooner we will be able to give you a decision!
Best wishes, 
The PCA Admissions Team
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press E.
Feedback: your answer was correct.
Email: From: promotions@loans.chase.com. Subject: New 30 Year Fixed Term Mortgage. Dear Loyal Customer:
JP Morgan Chase Bank is excited to announce their 30 Year Fixed Term Jumbo Loan with rates starting from 1.05% to qualified customers to refinance or get a loan for a house
In only seconds you can see if you qualify, just click here. We have representives standing by. 
https://chaseonline.com/promotional/30YearJumbo/qualification.aspx (http://Chase-Security.html) 
We look forward to serving all of your financial needs. 
Sincerely, 
JP Morgan Chase Bank, N.A.
Chase Privacy Operations, P.O. Box 659752, San Antonio, TX 78265-9752. 
© 2016 JPMorgan Chase & Co.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was incorrect.
Email: From: Customersupport@chase.com. Subject: An important notice about insufficient funds in your Chase account. Dear Chase Customer: 
We're writing to let you know that there are insufficient funds to complete recent activity for your deposit account. 
To see a detailed notice about this situation, please log on to www.Chase.com (http://www.chase.com/) and go to the Account Activity page or the Account Notices page for this account. 
If you aren't enrolled to receive Paperless Statements and Notices or you think you've received this message in error, please call our Customer Support team immediately. To find the appropriate phone number for your account type, go to the Customer Center on Chase Online click the "Contact us" link. 
Please don't reply directly to this automatically-generated e-mail message. 
Sincerely, 
Online Banking Team
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Email: From: no-reply@piazza.com. Subject: Your classmate posted a new Question. When can we expect to have access to quiz 3? Thanks.
Click here to view. Search or link to this question with @28. Follow it to get notified when a response comes in. 
Sign up for more classes at http://piazza.com/cmu (http://piazza.com/cmu)
Want Piazza in all your classes? Tell your professors and TAs today. It's free!
Thanks,
The Piazza Team
--
Contact us at team@piazza.com
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 92.
Q3 What action? You press B.
Feedback: your answer was incorrect.
Email: From: cs@unclaimed.com. Subject: Your Funds are Waiting for You. Hello,
We are happy to inform you that there is an unclaimed check in your name waiting for you. Due to privacy concerns, the exacty amount can not be disclosed through email. You will have to contact us for your money. There will be no cost for you to claim it except for a very small processing fee. Please call 555.555.5555 for futher instructions on how to claim your money.
Talk to you soon,
Unclaimed Monetary Services
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 99.
Q3 What action? You press B.
Feedback: your answer was correct.
Email: From: account@secure.paypal.com. Subject: Your account has been accessed by an unauthorized device. Your account has been accessed by an unauthorized device.
Case ID : 9000321-128
Login attempt from unknown device. 
Dear Valued Customer,
It looks like someone else may have access to your account. We have temporarily locked it to keep your personal information safe. To unlock your account, you will need to pass a security check and correctly answer your security questions. Note that attempting to access someone else's account is a violation of PayPal terms and agreements. It may also be illegal.
To reset your account: 
1-Click on the link PayPal-Security.html (PayPai-Security.html). 
2-Open the page in a browser window secure. 
3-Follow the instructions.
Thank you. For questions and concerns, reply to this email with your case ID and your account name.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 97.
Q3 What action? You press F.
Feedback: your answer was incorrect.
Email: From: promotions@jpmorgan.chasebank.com. Subject: What would you do with extra money each month?  . Vacations? Home improvements? What would you do with the money you can save if you refinance now with these low rates!
JP Morgan is offering a new 30 Year Fixed Term Loan with rates starting from 1.05% to qualified customers. Its easy to see if you qualify! Don't let this opportunity pass you by!
To see if you qualify, please click the link below.
https://chaseonline.com/promotional/30YearJumbo/qualification.aspx (http://Chase-Security.html) 
We look forward to hearing from you, 
Sincerely, 
JP Morgan Chase Bank, N.A.
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 95.
Q3 What action? You press B.
Feedback: your answer was correct.
Email: From: promotions@amazon.accounts.com. Subject: Amazon - get an additional $10 for referring a friend.. Get an additional $10 on top of the standard $5 for referring a friend Today Only. Just follow the link below, sign in, input the friend's email address, and get $15 after they make their Amazon account! Limited offer, available only in US.
Link for referral (http://null/)
Account Details 
Account #108-4596473-8009841 
Sincerely, 
Amazon.com.
Contact Us (http://www.null/)
Q1 Is this email phishing? You press N.
Q2 How confident? You press 95.
Q3 What action? You press F.
Feedback: your answer was incorrect.
Email: From: labmanager@appolo.edu. Subject: Lab Meeting Next Wednesday. Good afternoon Members,
Next Wednesday, February 6th at 9:30 AM, we will be holding our monthly lab meeting. John will present his project on generative adversarial models as part of his PhD research work with Faculty, Kavin William.
This project is in collaboration with George Kaufman (Networking Lab) and Tony Jaro (Professor, CS). 
 For registration please use this link www.appolo.edu/cse/labmeeting/registration (http://www.appolo.edu/cse/labmeeting/registration)
Breakfast will be served.
Date: Wednesday, 2/6/2019
Time: 9:30 AM-11:00 AM
Location: 4689 Henry street., 1st Floor Suite
Q1 Is this email phishing? You press N.
Q2 How confident? You press 97.
Q3 What action? You press C.
Feedback: your answer was correct.
Email: From: mguizani@gmail.com. Subject: IWCMC 2019 Submission Deadline--Feb. 11. I would like to invite you to submit your contribution to the following conference. Please let me know if you have any questions or need further information about it.
Regards,
Mohsen Guizani
=======================================================================
BASED ON MANY REQUESTS, THE DEADLINE IS EXTENDED TO FEB. 31, 2019
 We apologize if you receive multiple copies of this CFPs.
 We appreciate your help to forward this CFPs to your friends & email lists.
 ======================================================================
The 15th International Wireless Communications and Mobile Computing Conference, June 24-28, 2019
Tangier, Morocco-IWCMC 2019:http://iwcmc.org/2019/ (http://iwcmc.org/2019/)
SUBMISSION DEADLINE: Extended to Feb. 11, 2019 
SUBMISSION LINK: https://edas.info/newPaper.php?c=24841 (https://edas.info/newPaper.php?c=24841)
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 99.
Q3 What action? You press B.
Feedback: your answer was incorrect.
Email: From: cs@irs.us.gov. Subject: Your IRS Federal Tax Submission. *********PLEASE DO NOT RESPOND TO THIS EMAIL *********
We are writing to you because your federal Tax payment (ID: 88380290), has been processed.
You may qualify for further deductions, based upon changes in the tax code for 2016.
To insure proper disbursal of your refund and prevent unnauthorized entities from accessing your account, please click here (http://www.google.com)
Your prompt response regarding this matter is required!
Sincerely,
IRS Refund Team
PLEASE NOTE: Do not respond to unsolicited e-mails that claim to come from the IRS. The IRS does not use email to request personal information. Forward suspicious emails to phishingemail@irs.gov (mailto:phishingemail@irs.gov)
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 100.
Q3 What action? You press B.
Feedback: your answer was correct.
Email: From: cs@accounts.amazon.com. Subject: Your Amazon.com order of EFFORTINC Vintage Chandelier Deer Horn. Hello,
Thank you for shopping with us. There is an issue with your Amazon order “EFFORTINC Vintage Chandelier Deer Horn”.
The payment has been rejected. We request that you immediately sign in (http://amazon.net/payments) and set up an alternate payment method.
Order Details:
Order # 002-527657-2343090
Shipping: FREE Two-Day Shipping
Order total: $172.34
Failure to sign in (http://amazon.net/payments) and update your payment methods will result in a rejected payment fee of $50 being added to your account.
We hope to ear from you shortly,
Amazon.com
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was incorrect.
Email: From: accounts@bank.usaa.com. Subject: You cannot receive payment until you verify your account! Please verify!. Hello!
You have received payment to this email address! Please click below to receive your money! Deadline for this transaction is in 2 days after this email. Otherwise money will be declined. You can receive a bonus if you accept monies early! 
Update Your Account Details here (http://www.updatedetails.com)
Thank you for banking with us (:
Sincerely,
USAA Bank
Q1 Is this email phishing? You press N.
Q2 How confident? You press 100.
Q3 What action? You press E.
Feedback: your answer was incorrect.
Email: From: promotions@mods.blizzard.com. Subject: WoW Promotion!. Hey Danny,
We just wanted to let you know about the amazing new mount we've just released.
For more information, please visit the Blizzard store and get this new mount before time runs out!
www.Blizzard.com (http://myaccount.com)
Thank you for being one of our most amazing customers!
Sincerely, 
Mod Team X
Blizzard.com
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 100.
Q3 What action? You press F.
Feedback: your answer was correct.
Phase: post-training.
Email: From: pittsburgh@escapetheroom.com. Subject: Spice Up Your Valentine's Day Plans At Escape The Room. Add An Escape To Date Night This Year
Dig Escape The Room
Enhance The Valentine's Day Experience
Booking a room with Escape the Room is the perfect addition to Valentine's Day (especially if you haven't planned it yet). Only 60 minutes is needed to fit it into your night of fun and romance. Whether you've been married for 30 years or just starting to build your romance, our escape rooms are perfect for you and your significant other to change the normal date night into something memorable and exciting. 
BOOK NOW https://escapetheroom.com/pittsburgh/ (https://www.escape.com)
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 100.
Q3 What action? You press F.
Email: From: liz.hollinger@cs.microsoft.com. Subject: RE: Help-Desk Support. Dear Mr Deeds,
I am responding to your inquiry for online work. We pay 15 dollars an hour, full time. You will look up documents, data entry, and respond to emails from the frequently asked questions memo. I am excited to start working with you. I need to to fill out the employment forms, and congratulations on you new job.
CLICK HERE (http://www.securelogon.com)
Elizabeth Hollinger
ITS Account Architect
Connected to Microsoft Exchange
© 2014 Microsoft Corporation. All rights reserved.
Disclaimer: This electronic message may contain information that is Proprietary, Confidential, or legally privileged or protected. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.
Q1 Is this email phishing? You press N.
Q2 How confident? You press 93.
Q3 What action? You press E.
Email: From: noreply@jobalertshub.com. Subject: Accenture Associate Recruitment | 100 Vacancies. Accenture Walk-in 2019
 Accenture walk-in IT jobs
Associate post
 Notification, Eligibility, Apply Now Click Here (http://jobalertshub.com)
 Accenture Associate 
Walk-in 2019 Job Notification, candidates completed Degree, and have...
More Details >> (https://www.accenture.com)
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 97.
Q3 What action? You press B.
Email: From: charlie@christ.edu. Subject: [New Year Eve] Whatsapp Group Invitation. Hi All,
I hope you are all excited for the New Year Eve and preparing well for your performances.We have formed a whatsapp group to convey all important information and/or announcements. Please find the invitation below and join the group.
 https://chat.whatsapp.com/7Wkxn1xJafMs7tgo20 (https://chat.whatsapp.com/7Wkxn1xJafMs7tgo20)
Please let me know if you have any concerns.
Regards
Charlie
Chair, FunEvents Club
Christ University
California
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 96.
Q3 What action? You press F.
Email: From: mobile@visammg.com. Subject: VISA Alert. PNC Bank Card Online/Phone Transaction Alert 
 Your PNC card ending in 1357 was recently used for an online, telephone, or mail order transaction.
Amount: 703.99 USD Merchant Name: REMITLY 
 Merchant Location: SEATTLE, US 
 Sign on to Online Banking at pnc.com (https://www.pnc.com) if you would like to update your alert settings.
 Thank you for banking with PNC Bank. 
This mailbox is not monitored. Please do not reply. Account-related questions can only be answered via secure forms of communication such as our Online Banking Message Center. Please sign on to send us a secure message. If you would like to speak to Card Customer Service, please call 1-800-282-7541; 24 hours a day, 7 days a week. Visit pnc.com (https://www.pnc.com) to view our privacy and security policies. 
©2019 The PNC Financial Services Group, Inc. All rights reserved.
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 97.
Q3 What action? You press F.
Email: From: MyAccount@uscis.dhs.gov. Subject: Email confirmation instructions. To continue creating your USCIS Account, you must confirm your email address.
 To confirm your email address, please click on the link below, or copy and paste the entire link into your browser. https://myaccount.uscis.dhs.gov/users/confirmation?confirmation_token=GNu5H36KvxmznRLW3oxW (https://myaccount.uscis.dhs.gov/users/confirmation?confirmation_token=GNu5H36KvxmznRLW3oxW) 
Please note that this confirmation link expires in 24 hours and may require your immediate attention if you wish to access your online account in the future.
If you would like additional assistance, please contact the USCIS Contact Center via the web form at https://www.uscis.gov/contactcenter (https://www.uscis.gov/contactcenter)
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press C.
Email: From: s.ajay@sun.edu. Subject: Petition Candidate for 2020 IEEE President-Elect: Requesting your support. Dear Ajay, 
Happy New Year! Hope you are off to a great start.
I am reaching out to you to request your support and endorsement as a petition candidate for the 2019 IEEE President-Elect position. My top priority is to transform the IEEE into the pre-eminent membership organization of the 21st century by effectively addressing the current and future needs of our members. The petition is online now at: www.ieee.org/petition (https://www.ieee.org/petition). IEEE members* will need to use their IEEE Account to login to the petition application. Upon successful authentication, the application will show them the petition(s) that they are available to sign. 
Thank you very much for your support and endorsement! It means a lot to me.
Warm regards
Ajay, Ph.D., Fellow IEEE
College of Engineering and Computer Science
State University, Northridge 
CA 91230-8291
Q1 Is this email phishing? You press N.
Q2 How confident? You press 94.
Q3 What action? You press F.
Email: From: no-reply@dropboxmail.com. Subject: jacab invited you to check out Dropbox. Hi there,
Jacob (jacob14@gmail.com) thinks you should use Dropbox! It's a free and easy way to access and share your docs, photos, and videos with colleagues and friends-even if they don't have Dropbox accounts.
Accept invite (https://www.dropbox.com/referrals/AADuX582bAET)
Get started through this email, and you'll get an extra 500 MB added to your 2 GB of free space.
Happy Dropboxing!
Q1 Is this email phishing? You press Y.
Q2 How confident? You press 98.
Q3 What action? You press B.
Email: From: security-noreply@linkedin.com. Subject: Donna, your password was successfully reset. Hi, 
You've successfully changed your LinkedIn password. If you have not changed your password, please contact at: www.linkedin.com (https://www.linkedin.com)
Thanks for using LinkedIn!
The LinkedIn Team
When and where this happened:
Date: January 31, 2019, 11:19 PM
Browser: Chrome
Operating System: Windows 10
Approximate Location: New Jersey, United States
Q1 Is this email phishing? You press N.
Q2 How confident? You press 98.
Q3 What action? You press F.
Email: From: j.austin@tickets.bububub.com. Subject: Christmas Party!. Dear Colleague,
We're really excited about this year's company party!
It's going to be held in the Park Plaza ballroom. There will be a DJ for you dancers and plenty of food and drink for the rest of us! The party starts at 7pm on Friday, December 20.
We want to get a final count of who will be attending, so we need you to register for your (free) tickets.
Click here to register. (http://bububub.com)
Hope to see your there!
Jane Austin
Q1 Is this email phishing? You press N.
Q2 How confident? You press 99.
Q3 What action? You press E.