It’s very interesting to see the intuitive approach here and there is a lot to like about how you identified something you didn’t like in some personality tests (though there are some concrete ones out there), probed content domains for item generation, and settled upon correlations to assess hanging-togetherness.
But you need to incorporate your knowledge from reading about scale development and factor analysis. Obviously you’ve read in that space. You know you want to test item-total correlations (trait impact), multi-dimensionality (factor model loss), and criterion validity (correlation with lexical notion). Are you trying to ease us in with a primer (with different vocabulary!) or reinvent the wheel?
Let’s start with the easy-goingness scale:
(+) In the evening I tend to relax and watch some videos/TV
(+) I don’t feel the need to arrange any elaborate events to go to in my free time
(+) I think it is best to take it easy about exams and interviews, rather than worrying a bunch about doing it right
(+) I think you’ve got to have low expectations of others, as otherwise they will let you down
(-) I get angry about politics
(-) I have a stressful job
(-) I don’t feel like I should have breaks at work unless I’ve “earned” them by finishing something productive
(-) I spent a lot of effort on parenting
The breadth of it is either a strength or a weakness. It’d be nice to have a construct definition or at least some gesturing at what easy-goingness actually is to gauge the face-validity of these items. Concrete items necessarily will have some domain-dependence, resulting in deficiency (e.g., someone who likes to relax and read a book will score low on item 1) or contamination (e.g., having low expectations of others might also be trait pessimism), but item 8 is really specific. It hampers the ability of this scale to capture easy-goingness among non-parents. The breadth would be good if it captured variations on easy-goingness, but instead it’d be bad if it just captures different things that don’t really relate to each other. That’s especially problematic because then the inference from low inter-correlations might not be that the construct is bad, but that the items just don’t tap into it. You can see where I’m going with this because...
This suggests to me that Easy-Goingness is not very “real”. While it might make sense to describe a person as doing something Easy-Going, for instance when they are watching TV, it is kind of arbitrary to talk about people as being more or less Easy-Going, because it depends a lot on context/what you mean.
...indeed, the items are mainly just capturing different things, not reflecting on easy-goingness in any way. From a scale-assessment standpoint, it’s great to see the results confirm my unease about the items based on simply reading them.
The fact that this is weak means that even the most Easy-Going people cannot necessarily be expected to be particularly Easy-Going in all contexts.
This statement presumes your measure reflects a higher-order easy-goingness and that context-specific easy-goingnesses are also being adequately measured.
With conservatism, on the other hand, you can see there is some context-specificity (e.g., dress vs. general social views vs. issue-based ideology), but the measure is facially better. And it hangs together better. Alternately, you might explore those contours and say you’ve come up with a multi-dimensional conservatism scale, just like you have a multi-dimensional creativity scale.
the “Correlation with lexical notion” was consistently close to 1, showing that the concrete and the abstract descriptors were getting at the same thing.
There’s an implicit “when the concrete descriptors actually had face validity” hidden here; low correlation with the lexical notion could indicate a problem with the lexical scale or a problem with the concrete scale, or both.
Overall, I am very impressed that you presented a scary chart to start, promised you’d explain it, and successfully did so. The general takeaway from it is that the lexical hypothesis could be pretty sound and a few of these might be multidimensional in nature (or could be that some items are good and some a bad). For the low trait impact scales, it’s a question of whether the items are good and the construct isn’t “real,” or whether the items are just a bad measurement approach.
But you need to incorporate your knowledge from reading about scale development and factor analysis. Obviously you’ve read in that space. You know you want to test item-total correlations (trait impact), multi-dimensionality (factor model loss), and criterion validity (correlation with lexical notion). Are you trying to ease us in with a primer (with different vocabulary!) or reinvent the wheel?
Good question. In retrospect, I should probably have put more effort into using standard terms. That said:
Test item-total correlations: Strictly speaking “factor loadings” would be a better term, since I did not compute it based on a correlation with a test score, but instead with a CFA-style factor model.
Multidimensionality: Maybe. Obviously it’s multidimensionality that I am trying to test, but literally my score for the tests is a least-squares loss for a CFA-style factor model.
Criterion validity: Maybe. Arguably convergent/concurrent validity would be even more standard terms. But I think “Correlation with lexical notion” is more specific.
The breadth of it is either a strength or a weakness. It’d be nice to have a construct definition or at least some gesturing at what easy-goingness actually is to gauge the face-validity of these items.
The items are each meant to assess something from the stories I collected from someone who empirically scored high and low on easy-goingness scales. So their validity criterion is not meant to be in assessing easy-goingness generally, but in assessing the thing from those stories. Here are the stories corresponding to each item:
In the evening I tend to relax and watch some videos/TV
When I finish work for the day I often go straight home and jump into my pyjamas. I like to relax and watch some tv and films to unwind after a long day—usually with a glass of wine. Certain days when I come home my partner would like to travel for a couple hours to go dog walking and enjoying time outside. No matter what kind of day I have at work I am always keen to do anything my partner/family/friends would like to do as is in my nature.
(+) I don’t feel the need to arrange any elaborate events to go to in my free time
I think I dont need to always go out in evenings to feel socially connected. Rather I would sit and enjoy the quiet at home. Moreover I dont get easily flustered if people have different opinions compared to me. I dont easily get offended and can take things in a right spirit. so, I am easy to approach
(+) I think it is best to take it easy about exams and interviews, rather than worrying a bunch about doing it right
I think youy have to be going in life otherwise everything will get to to. For example when i did my exams at uni and school, you have to be easy going to cope with the stress and fear thatr comes with them. This can be applied to anything though, if you are not easy going the littel things will get to you and you will have no chnace being able to cope with the big issues in life.
(+) I think you’ve got to have low expectations of others, as otherwise they will let you down
I am easy going in that I do not have high expectations of others because I have learnt that people let you down and if there was no expectation in the first place you cannot be surprised or disappointed, on the other hand if you expect nothing you can be quite pleasantly surprised. I always try to see both sides of any argument or situation and consider that everyone has the right to an opinion that does not have to match my own.
(-) I get angry about politics
I was in a team dinner party and in a discussion about politics which I joined in with other colleuages. There was a lot of talk about dealing with education, the economy and how to restore the leadership of the labour party back then and the Iraq war all of which I was onboard with . then came questions about what to do with flooding immigrants and how to control them, given my uncles both were illegal immigratns back then but managed to claw citizenship after 10 years I was uneasy joining the discussion and there was a lot of talk on what races were the culprits. I said only legal immigration should be allowed but did not join further focusing on my drink instead knowing the discussion was a race hate discussion and I was indirectly being attacked. Next 10 mins I made up an excuse to leave and left the party but faked goodbyes but was angry that I had to work with scumbag colleagues.
(-) I have a stressful job
(-) I spent a lot of effort on parenting
My life is not at all leisurely. I have two small children and a stressful job. If I were to be easy going about everything things wouldn’t get done and our lives would feel chaotic. There needs to be a balance between being easy going and highly strung. I don’t like to forget things that need doing or let people down.
(-) I don’t feel like I should have breaks at work unless I’ve “earned” them by finishing something productive
A simple example is that when I arrive at work, my boss often asks at once if I want a coffee, as he often wants one at the beginning of the day. I prefer to do some work before having a coffee, as to me it signifies a moment of relaxation and to the puritan work ethic part of me, it doesn’t make sense to have a break until I have “earned” it.
There’s definitely a lot from these stories that I fail to capture. Often the participants mention multiple things and I only ask about one of them. I could easily imagine the items could be made better.
item 8 is really specific. It hampers the ability of this scale to capture easy-goingness among non-parents.
Maybe.
Really in the general population, most people are parents, so I don’t think it is much more specific than the other items. But my respondents skew quite young, so it is probably a problem for my sample. Might be interesting to add an interaction model to this later though.
This statement presumes your measure reflects a higher-order easy-goingness and that context-specific easy-goingnesses are also being adequately measured.
With conservatism, on the other hand, you can see there is some context-specificity (e.g., dress vs. general social views vs. issue-based ideology), but the measure is facially better. And it hangs together better. Alternately, you might explore those contours and say you’ve come up with a multi-dimensional conservatism scale, just like you have a multi-dimensional creativity scale.
🤷 I constructed the conservatism and easy-goingness items in the same way, so I think there is something inherent to conservatism that makes it cohere more than easy-goingness.
There’s an implicit “when the concrete descriptors actually had face validity” hidden here; low correlation with the lexical notion could indicate a problem with the lexical scale or a problem with the concrete scale, or both.
I think of it as an empirical test of the concrete descriptor’s validity. That is, the abstract predictors have face validity, and if these are highly correlated with the concrete descriptors, then at least we know the concrete descriptors are not measuring anything other than the traits they are intended to measure.
It’s very interesting to see the intuitive approach here and there is a lot to like about how you identified something you didn’t like in some personality tests (though there are some concrete ones out there), probed content domains for item generation, and settled upon correlations to assess hanging-togetherness.
But you need to incorporate your knowledge from reading about scale development and factor analysis. Obviously you’ve read in that space. You know you want to test item-total correlations (trait impact), multi-dimensionality (factor model loss), and criterion validity (correlation with lexical notion). Are you trying to ease us in with a primer (with different vocabulary!) or reinvent the wheel?
Let’s start with the easy-goingness scale:
The breadth of it is either a strength or a weakness. It’d be nice to have a construct definition or at least some gesturing at what easy-goingness actually is to gauge the face-validity of these items. Concrete items necessarily will have some domain-dependence, resulting in deficiency (e.g., someone who likes to relax and read a book will score low on item 1) or contamination (e.g., having low expectations of others might also be trait pessimism), but item 8 is really specific. It hampers the ability of this scale to capture easy-goingness among non-parents. The breadth would be good if it captured variations on easy-goingness, but instead it’d be bad if it just captures different things that don’t really relate to each other. That’s especially problematic because then the inference from low inter-correlations might not be that the construct is bad, but that the items just don’t tap into it. You can see where I’m going with this because...
...indeed, the items are mainly just capturing different things, not reflecting on easy-goingness in any way. From a scale-assessment standpoint, it’s great to see the results confirm my unease about the items based on simply reading them.
This statement presumes your measure reflects a higher-order easy-goingness and that context-specific easy-goingnesses are also being adequately measured.
With conservatism, on the other hand, you can see there is some context-specificity (e.g., dress vs. general social views vs. issue-based ideology), but the measure is facially better. And it hangs together better. Alternately, you might explore those contours and say you’ve come up with a multi-dimensional conservatism scale, just like you have a multi-dimensional creativity scale.
There’s an implicit “when the concrete descriptors actually had face validity” hidden here; low correlation with the lexical notion could indicate a problem with the lexical scale or a problem with the concrete scale, or both.
Overall, I am very impressed that you presented a scary chart to start, promised you’d explain it, and successfully did so. The general takeaway from it is that the lexical hypothesis could be pretty sound and a few of these might be multidimensional in nature (or could be that some items are good and some a bad). For the low trait impact scales, it’s a question of whether the items are good and the construct isn’t “real,” or whether the items are just a bad measurement approach.
Thank you for your in-depth response!
Good question. In retrospect, I should probably have put more effort into using standard terms. That said:
Test item-total correlations: Strictly speaking “factor loadings” would be a better term, since I did not compute it based on a correlation with a test score, but instead with a CFA-style factor model.
Multidimensionality: Maybe. Obviously it’s multidimensionality that I am trying to test, but literally my score for the tests is a least-squares loss for a CFA-style factor model.
Criterion validity: Maybe. Arguably convergent/concurrent validity would be even more standard terms. But I think “Correlation with lexical notion” is more specific.
The items are each meant to assess something from the stories I collected from someone who empirically scored high and low on easy-goingness scales. So their validity criterion is not meant to be in assessing easy-goingness generally, but in assessing the thing from those stories. Here are the stories corresponding to each item:
In the evening I tend to relax and watch some videos/TV
(+) I don’t feel the need to arrange any elaborate events to go to in my free time
(+) I think it is best to take it easy about exams and interviews, rather than worrying a bunch about doing it right
(+) I think you’ve got to have low expectations of others, as otherwise they will let you down
(-) I get angry about politics
(-) I have a stressful job
(-) I spent a lot of effort on parenting
(-) I don’t feel like I should have breaks at work unless I’ve “earned” them by finishing something productive
There’s definitely a lot from these stories that I fail to capture. Often the participants mention multiple things and I only ask about one of them. I could easily imagine the items could be made better.
Maybe.
Really in the general population, most people are parents, so I don’t think it is much more specific than the other items. But my respondents skew quite young, so it is probably a problem for my sample. Might be interesting to add an interaction model to this later though.
🤷 I constructed the conservatism and easy-goingness items in the same way, so I think there is something inherent to conservatism that makes it cohere more than easy-goingness.
I think of it as an empirical test of the concrete descriptor’s validity. That is, the abstract predictors have face validity, and if these are highly correlated with the concrete descriptors, then at least we know the concrete descriptors are not measuring anything other than the traits they are intended to measure.